2023-02-10 13:53:57,345 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure timeout: 13 mins 2023-02-10 13:53:57,581 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c 2023-02-10 13:53:57,597 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-10 13:53:57,597 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8, deleteOnExit=true 2023-02-10 13:53:57,597 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-10 13:53:57,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/test.cache.data in system properties and HBase conf 2023-02-10 13:53:57,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/hadoop.tmp.dir in system properties and HBase conf 2023-02-10 13:53:57,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/hadoop.log.dir in system properties and HBase conf 2023-02-10 13:53:57,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-10 13:53:57,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-10 13:53:57,599 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-10 13:53:57,709 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-10 13:53:58,094 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-10 13:53:58,100 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-10 13:53:58,101 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-10 13:53:58,101 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-10 13:53:58,102 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-10 13:53:58,102 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-10 13:53:58,103 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-10 13:53:58,103 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-10 13:53:58,103 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-10 13:53:58,104 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-10 13:53:58,104 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/nfs.dump.dir in system properties and HBase conf 2023-02-10 13:53:58,105 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/java.io.tmpdir in system properties and HBase conf 2023-02-10 13:53:58,105 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-10 13:53:58,106 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-10 13:53:58,106 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-10 13:53:58,658 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-10 13:53:58,663 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-10 13:53:59,760 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-10 13:53:59,929 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-10 13:53:59,944 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-10 13:53:59,982 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-10 13:54:00,015 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/java.io.tmpdir/Jetty_localhost_39143_hdfs____798ir8/webapp 2023-02-10 13:54:00,181 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39143 2023-02-10 13:54:00,192 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-10 13:54:00,192 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-10 13:54:00,625 WARN [Listener at localhost/36875] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-10 13:54:00,691 WARN [Listener at localhost/36875] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-10 13:54:00,708 WARN [Listener at localhost/36875] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-10 13:54:00,714 INFO [Listener at localhost/36875] log.Slf4jLog(67): jetty-6.1.26 2023-02-10 13:54:00,718 INFO [Listener at localhost/36875] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/java.io.tmpdir/Jetty_localhost_33539_datanode____lc26hu/webapp 2023-02-10 13:54:00,819 INFO [Listener at localhost/36875] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33539 2023-02-10 13:54:01,138 WARN [Listener at localhost/40333] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-10 13:54:01,164 WARN [Listener at localhost/40333] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-10 13:54:01,169 WARN [Listener at localhost/40333] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-10 13:54:01,170 INFO [Listener at localhost/40333] log.Slf4jLog(67): jetty-6.1.26 2023-02-10 13:54:01,176 INFO [Listener at localhost/40333] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/java.io.tmpdir/Jetty_localhost_44523_datanode____1pth25/webapp 2023-02-10 13:54:01,274 INFO [Listener at localhost/40333] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44523 2023-02-10 13:54:01,284 WARN [Listener at localhost/36045] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-10 13:54:01,312 WARN [Listener at localhost/36045] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-10 13:54:01,316 WARN [Listener at localhost/36045] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-10 13:54:01,317 INFO [Listener at localhost/36045] log.Slf4jLog(67): jetty-6.1.26 2023-02-10 13:54:01,325 INFO [Listener at localhost/36045] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/java.io.tmpdir/Jetty_localhost_44891_datanode____rovu0n/webapp 2023-02-10 13:54:01,450 INFO [Listener at localhost/36045] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44891 2023-02-10 13:54:01,463 WARN [Listener at localhost/43477] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-10 13:54:01,656 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4c961603c73338fc: Processing first storage report for DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f from datanode 10b8853b-22ce-4aa3-86bd-2e3e9072bae3 2023-02-10 13:54:01,657 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4c961603c73338fc: from storage DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f node DatanodeRegistration(127.0.0.1:40385, datanodeUuid=10b8853b-22ce-4aa3-86bd-2e3e9072bae3, infoPort=35419, infoSecurePort=0, ipcPort=40333, storageInfo=lv=-57;cid=testClusterID;nsid=1001986124;c=1676037238735), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-10 13:54:01,658 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xef5fdd0bf28083b7: Processing first storage report for DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8 from datanode 7cac6cd9-03f4-49d1-b43b-1428b7e6fbc3 2023-02-10 13:54:01,658 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xef5fdd0bf28083b7: from storage DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8 node DatanodeRegistration(127.0.0.1:39211, datanodeUuid=7cac6cd9-03f4-49d1-b43b-1428b7e6fbc3, infoPort=41635, infoSecurePort=0, ipcPort=43477, storageInfo=lv=-57;cid=testClusterID;nsid=1001986124;c=1676037238735), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-10 13:54:01,658 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf8be145b962d6b78: Processing first storage report for DS-17591b7b-de3e-43bb-863b-f2e0e725e205 from datanode 3707987e-c8ef-479c-bbb7-c70ee662bee7 2023-02-10 13:54:01,658 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf8be145b962d6b78: from storage DS-17591b7b-de3e-43bb-863b-f2e0e725e205 node DatanodeRegistration(127.0.0.1:36191, datanodeUuid=3707987e-c8ef-479c-bbb7-c70ee662bee7, infoPort=45399, infoSecurePort=0, ipcPort=36045, storageInfo=lv=-57;cid=testClusterID;nsid=1001986124;c=1676037238735), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-10 13:54:01,659 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4c961603c73338fc: Processing first storage report for DS-70fa3002-bc34-4b74-9229-5fca381c989f from datanode 10b8853b-22ce-4aa3-86bd-2e3e9072bae3 2023-02-10 13:54:01,659 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4c961603c73338fc: from storage DS-70fa3002-bc34-4b74-9229-5fca381c989f node DatanodeRegistration(127.0.0.1:40385, datanodeUuid=10b8853b-22ce-4aa3-86bd-2e3e9072bae3, infoPort=35419, infoSecurePort=0, ipcPort=40333, storageInfo=lv=-57;cid=testClusterID;nsid=1001986124;c=1676037238735), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-10 13:54:01,659 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xef5fdd0bf28083b7: Processing first storage report for DS-1c225aab-cc52-4a91-be13-62afbcfda532 from datanode 7cac6cd9-03f4-49d1-b43b-1428b7e6fbc3 2023-02-10 13:54:01,659 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xef5fdd0bf28083b7: from storage DS-1c225aab-cc52-4a91-be13-62afbcfda532 node DatanodeRegistration(127.0.0.1:39211, datanodeUuid=7cac6cd9-03f4-49d1-b43b-1428b7e6fbc3, infoPort=41635, infoSecurePort=0, ipcPort=43477, storageInfo=lv=-57;cid=testClusterID;nsid=1001986124;c=1676037238735), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-10 13:54:01,659 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf8be145b962d6b78: Processing first storage report for DS-7e578ce6-2f93-4da5-934d-26ec65e1ac4e from datanode 3707987e-c8ef-479c-bbb7-c70ee662bee7 2023-02-10 13:54:01,660 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf8be145b962d6b78: from storage DS-7e578ce6-2f93-4da5-934d-26ec65e1ac4e node DatanodeRegistration(127.0.0.1:36191, datanodeUuid=3707987e-c8ef-479c-bbb7-c70ee662bee7, infoPort=45399, infoSecurePort=0, ipcPort=36045, storageInfo=lv=-57;cid=testClusterID;nsid=1001986124;c=1676037238735), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-10 13:54:01,879 DEBUG [Listener at localhost/43477] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c 2023-02-10 13:54:01,945 INFO [Listener at localhost/43477] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/zookeeper_0, clientPort=63999, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-10 13:54:01,959 INFO [Listener at localhost/43477] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63999 2023-02-10 13:54:01,966 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:01,968 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:02,620 INFO [Listener at localhost/43477] util.FSUtils(479): Created version file at hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963 with version=8 2023-02-10 13:54:02,620 INFO [Listener at localhost/43477] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/hbase-staging 2023-02-10 13:54:02,942 INFO [Listener at localhost/43477] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-10 13:54:03,385 INFO [Listener at localhost/43477] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-02-10 13:54:03,417 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,417 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,418 INFO [Listener at localhost/43477] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-10 13:54:03,418 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,418 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-10 13:54:03,554 INFO [Listener at localhost/43477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-10 13:54:03,623 DEBUG [Listener at localhost/43477] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-10 13:54:03,712 INFO [Listener at localhost/43477] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38205 2023-02-10 13:54:03,722 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:03,724 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:03,746 INFO [Listener at localhost/43477] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38205 connecting to ZooKeeper ensemble=127.0.0.1:63999 2023-02-10 13:54:03,788 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:382050x0, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-10 13:54:03,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38205-0x1003e82a6100000 connected 2023-02-10 13:54:03,816 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-10 13:54:03,816 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:54:03,820 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-10 13:54:03,827 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38205 2023-02-10 13:54:03,827 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38205 2023-02-10 13:54:03,828 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38205 2023-02-10 13:54:03,828 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38205 2023-02-10 13:54:03,829 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38205 2023-02-10 13:54:03,834 INFO [Listener at localhost/43477] master.HMaster(439): hbase.rootdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963, hbase.cluster.distributed=false 2023-02-10 13:54:03,911 INFO [Listener at localhost/43477] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-10 13:54:03,912 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,912 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,912 INFO [Listener at localhost/43477] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-10 13:54:03,912 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,913 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-10 13:54:03,919 INFO [Listener at localhost/43477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-10 13:54:03,922 INFO [Listener at localhost/43477] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39591 2023-02-10 13:54:03,924 INFO [Listener at localhost/43477] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-10 13:54:03,938 DEBUG [Listener at localhost/43477] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-10 13:54:03,939 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:03,942 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:03,945 INFO [Listener at localhost/43477] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39591 connecting to ZooKeeper ensemble=127.0.0.1:63999 2023-02-10 13:54:03,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:395910x0, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-10 13:54:03,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39591-0x1003e82a6100001 connected 2023-02-10 13:54:03,951 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-10 13:54:03,953 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:54:03,954 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-10 13:54:03,955 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39591 2023-02-10 13:54:03,955 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39591 2023-02-10 13:54:03,961 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39591 2023-02-10 13:54:03,962 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39591 2023-02-10 13:54:03,962 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39591 2023-02-10 13:54:03,976 INFO [Listener at localhost/43477] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-10 13:54:03,976 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,976 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,977 INFO [Listener at localhost/43477] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-10 13:54:03,977 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:03,977 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-10 13:54:03,978 INFO [Listener at localhost/43477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-10 13:54:03,980 INFO [Listener at localhost/43477] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46307 2023-02-10 13:54:03,980 INFO [Listener at localhost/43477] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-10 13:54:03,981 DEBUG [Listener at localhost/43477] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-10 13:54:03,982 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:03,985 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:03,987 INFO [Listener at localhost/43477] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46307 connecting to ZooKeeper ensemble=127.0.0.1:63999 2023-02-10 13:54:03,990 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:463070x0, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-10 13:54:03,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46307-0x1003e82a6100002 connected 2023-02-10 13:54:03,992 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-10 13:54:03,993 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:54:03,995 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-10 13:54:03,996 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46307 2023-02-10 13:54:03,996 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46307 2023-02-10 13:54:03,997 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46307 2023-02-10 13:54:03,998 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46307 2023-02-10 13:54:03,998 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46307 2023-02-10 13:54:04,010 INFO [Listener at localhost/43477] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-02-10 13:54:04,010 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:04,011 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:04,011 INFO [Listener at localhost/43477] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-10 13:54:04,011 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-10 13:54:04,011 INFO [Listener at localhost/43477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-10 13:54:04,011 INFO [Listener at localhost/43477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-10 13:54:04,012 INFO [Listener at localhost/43477] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40651 2023-02-10 13:54:04,013 INFO [Listener at localhost/43477] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-10 13:54:04,014 DEBUG [Listener at localhost/43477] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-10 13:54:04,015 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:04,017 INFO [Listener at localhost/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:04,018 INFO [Listener at localhost/43477] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40651 connecting to ZooKeeper ensemble=127.0.0.1:63999 2023-02-10 13:54:04,022 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:406510x0, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-10 13:54:04,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40651-0x1003e82a6100003 connected 2023-02-10 13:54:04,023 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-10 13:54:04,023 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:54:04,024 DEBUG [Listener at localhost/43477] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-10 13:54:04,025 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40651 2023-02-10 13:54:04,025 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40651 2023-02-10 13:54:04,026 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40651 2023-02-10 13:54:04,026 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40651 2023-02-10 13:54:04,026 DEBUG [Listener at localhost/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40651 2023-02-10 13:54:04,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,037 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-10 13:54:04,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,057 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-10 13:54:04,057 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-10 13:54:04,057 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-10 13:54:04,057 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-10 13:54:04,058 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:04,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-10 13:54:04,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38205,1676037242772 from backup master directory 2023-02-10 13:54:04,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-10 13:54:04,065 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,065 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-10 13:54:04,066 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-10 13:54:04,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,068 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-10 13:54:04,070 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-10 13:54:04,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/hbase.id with ID: e6c03438-436a-4d84-a515-f347b8185cbf 2023-02-10 13:54:04,200 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-10 13:54:04,214 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:04,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x70bdf07e to 127.0.0.1:63999 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-10 13:54:04,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@303378f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-10 13:54:04,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-10 13:54:04,320 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-10 13:54:04,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-10 13:54:04,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-10 13:54:04,341 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-10 13:54:04,345 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-10 13:54:04,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-10 13:54:04,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store-tmp 2023-02-10 13:54:04,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:04,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-10 13:54:04,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:54:04,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:54:04,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-10 13:54:04,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:54:04,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:54:04,433 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676037244431Disabling compacts and flushes for region at 1676037244432 (+1 ms)Disabling writes for close at 1676037244432Writing region close event to WAL at 1676037244432Closed at 1676037244432 2023-02-10 13:54:04,434 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/WALs/jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,460 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38205%2C1676037242772, suffix=, logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/WALs/jenkins-hbase4.apache.org,38205,1676037242772, archiveDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/oldWALs, maxLogs=10 2023-02-10 13:54:04,517 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK] 2023-02-10 13:54:04,517 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK] 2023-02-10 13:54:04,517 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK] 2023-02-10 13:54:04,525 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-10 13:54:04,601 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/WALs/jenkins-hbase4.apache.org,38205,1676037242772/jenkins-hbase4.apache.org%2C38205%2C1676037242772.1676037244471 2023-02-10 13:54:04,602 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK], DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK]] 2023-02-10 13:54:04,602 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:04,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:04,606 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-10 13:54:04,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-10 13:54:04,674 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-10 13:54:04,681 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-10 13:54:04,704 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-10 13:54:04,716 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:04,722 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-10 13:54:04,724 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-10 13:54:04,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-10 13:54:04,742 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:04,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59800785, jitterRate=-0.10889886319637299}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-10 13:54:04,743 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676037244643Initializing all the Stores at 1676037244646 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037244646Cleaning up temporary data from old regions at 1676037244730 (+84 ms)Cleaning up detritus from prior splits at 1676037244731 (+1 ms)Region opened successfully at 1676037244743 (+12 ms) 2023-02-10 13:54:04,744 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-10 13:54:04,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:04,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:04,767 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:04,768 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-10 13:54:04,802 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-02-10 13:54:04,802 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-10 13:54:04,827 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-10 13:54:04,832 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-10 13:54:04,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-10 13:54:04,859 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-10 13:54:04,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-10 13:54:04,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-10 13:54:04,870 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-10 13:54:04,874 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:04,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-10 13:54:04,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-10 13:54:04,891 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-10 13:54:04,896 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-10 13:54:04,896 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-10 13:54:04,896 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-10 13:54:04,896 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:04,896 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-10 13:54:04,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,38205,1676037242772, sessionid=0x1003e82a6100000, setting cluster-up flag (Was=false) 2023-02-10 13:54:04,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:04,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-10 13:54:04,923 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,929 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:04,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-10 13:54:04,936 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:04,939 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.hbase-snapshot/.tmp 2023-02-10 13:54:05,031 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(952): ClusterId : e6c03438-436a-4d84-a515-f347b8185cbf 2023-02-10 13:54:05,032 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(952): ClusterId : e6c03438-436a-4d84-a515-f347b8185cbf 2023-02-10 13:54:05,047 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-10 13:54:05,048 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(952): ClusterId : e6c03438-436a-4d84-a515-f347b8185cbf 2023-02-10 13:54:05,056 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-10 13:54:05,049 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-10 13:54:05,061 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-10 13:54:05,061 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-10 13:54:05,062 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-10 13:54:05,062 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-10 13:54:05,062 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-10 13:54:05,062 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-10 13:54:05,067 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-10 13:54:05,067 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-10 13:54:05,076 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-10 13:54:05,076 DEBUG [RS:1;jenkins-hbase4:46307] zookeeper.ReadOnlyZKClient(139): Connect 0x4fc258b9 to 127.0.0.1:63999 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-10 13:54:05,082 DEBUG [RS:2;jenkins-hbase4:40651] zookeeper.ReadOnlyZKClient(139): Connect 0x0fd7b187 to 127.0.0.1:63999 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-10 13:54:05,082 DEBUG [RS:0;jenkins-hbase4:39591] zookeeper.ReadOnlyZKClient(139): Connect 0x19aceb18 to 127.0.0.1:63999 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-10 13:54:05,112 DEBUG [RS:1;jenkins-hbase4:46307] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4105bf31, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-10 13:54:05,113 DEBUG [RS:1;jenkins-hbase4:46307] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@453a33c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-10 13:54:05,114 DEBUG [RS:2;jenkins-hbase4:40651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1104e0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-10 13:54:05,115 DEBUG [RS:2;jenkins-hbase4:40651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52fdaf7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-10 13:54:05,117 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:05,117 DEBUG [RS:0;jenkins-hbase4:39591] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@176b065d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-10 13:54:05,117 DEBUG [RS:0;jenkins-hbase4:39591] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13548068, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-10 13:54:05,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-10 13:54:05,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-10 13:54:05,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-10 13:54:05,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-10 13:54:05,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-10 13:54:05,128 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-10 13:54:05,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037275129 2023-02-10 13:54:05,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-10 13:54:05,141 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:40651 2023-02-10 13:54:05,142 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39591 2023-02-10 13:54:05,145 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-10 13:54:05,148 INFO [RS:2;jenkins-hbase4:40651] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-10 13:54:05,148 INFO [RS:2;jenkins-hbase4:40651] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-10 13:54:05,148 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1023): About to register with Master. 2023-02-10 13:54:05,148 INFO [RS:0;jenkins-hbase4:39591] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-10 13:54:05,149 INFO [RS:0;jenkins-hbase4:39591] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-10 13:54:05,149 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:05,150 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1023): About to register with Master. 2023-02-10 13:54:05,150 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-10 13:54:05,153 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38205,1676037242772 with isa=jenkins-hbase4.apache.org/172.31.14.131:39591, startcode=1676037243910 2023-02-10 13:54:05,154 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:46307 2023-02-10 13:54:05,154 INFO [RS:1;jenkins-hbase4:46307] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-10 13:54:05,154 INFO [RS:1;jenkins-hbase4:46307] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-10 13:54:05,154 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1023): About to register with Master. 2023-02-10 13:54:05,154 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38205,1676037242772 with isa=jenkins-hbase4.apache.org/172.31.14.131:40651, startcode=1676037244010 2023-02-10 13:54:05,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-10 13:54:05,155 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38205,1676037242772 with isa=jenkins-hbase4.apache.org/172.31.14.131:46307, startcode=1676037243975 2023-02-10 13:54:05,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-10 13:54:05,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-10 13:54:05,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-10 13:54:05,158 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-10 13:54:05,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-10 13:54:05,170 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-10 13:54:05,171 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-10 13:54:05,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-10 13:54:05,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-10 13:54:05,181 DEBUG [RS:2;jenkins-hbase4:40651] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-10 13:54:05,183 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676037245180,5,FailOnTimeoutGroup] 2023-02-10 13:54:05,181 DEBUG [RS:1;jenkins-hbase4:46307] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-10 13:54:05,181 DEBUG [RS:0;jenkins-hbase4:39591] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-10 13:54:05,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676037245183,5,FailOnTimeoutGroup] 2023-02-10 13:54:05,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-10 13:54:05,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,186 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,252 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:05,254 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:05,254 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963 2023-02-10 13:54:05,273 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60649, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-10 13:54:05,273 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34667, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-10 13:54:05,274 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49967, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-10 13:54:05,320 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,321 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38205] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38205] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,329 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:05,332 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-10 13:54:05,335 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/info 2023-02-10 13:54:05,338 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-10 13:54:05,339 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:05,339 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-10 13:54:05,343 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/rep_barrier 2023-02-10 13:54:05,343 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963 2023-02-10 13:54:05,343 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963 2023-02-10 13:54:05,343 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963 2023-02-10 13:54:05,344 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36875 2023-02-10 13:54:05,344 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36875 2023-02-10 13:54:05,344 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-10 13:54:05,344 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-10 13:54:05,344 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36875 2023-02-10 13:54:05,344 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-10 13:54:05,345 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-10 13:54:05,345 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:05,346 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-10 13:54:05,348 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/table 2023-02-10 13:54:05,349 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-10 13:54:05,350 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:05,350 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:54:05,352 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740 2023-02-10 13:54:05,353 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740 2023-02-10 13:54:05,354 DEBUG [RS:2;jenkins-hbase4:40651] zookeeper.ZKUtil(162): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,354 WARN [RS:2;jenkins-hbase4:40651] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-10 13:54:05,354 INFO [RS:2;jenkins-hbase4:40651] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-10 13:54:05,354 DEBUG [RS:0;jenkins-hbase4:39591] zookeeper.ZKUtil(162): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,354 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,355 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40651,1676037244010] 2023-02-10 13:54:05,355 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46307,1676037243975] 2023-02-10 13:54:05,354 WARN [RS:0;jenkins-hbase4:39591] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-10 13:54:05,355 DEBUG [RS:1;jenkins-hbase4:46307] zookeeper.ZKUtil(162): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,355 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39591,1676037243910] 2023-02-10 13:54:05,356 WARN [RS:1;jenkins-hbase4:46307] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-10 13:54:05,355 INFO [RS:0;jenkins-hbase4:39591] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-10 13:54:05,356 INFO [RS:1;jenkins-hbase4:46307] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-10 13:54:05,357 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,357 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,362 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-10 13:54:05,365 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-10 13:54:05,369 DEBUG [RS:1;jenkins-hbase4:46307] zookeeper.ZKUtil(162): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,369 DEBUG [RS:2;jenkins-hbase4:40651] zookeeper.ZKUtil(162): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,369 DEBUG [RS:0;jenkins-hbase4:39591] zookeeper.ZKUtil(162): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,370 DEBUG [RS:2;jenkins-hbase4:40651] zookeeper.ZKUtil(162): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,370 DEBUG [RS:0;jenkins-hbase4:39591] zookeeper.ZKUtil(162): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,370 DEBUG [RS:1;jenkins-hbase4:46307] zookeeper.ZKUtil(162): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,371 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:05,371 DEBUG [RS:2;jenkins-hbase4:40651] zookeeper.ZKUtil(162): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,371 DEBUG [RS:1;jenkins-hbase4:46307] zookeeper.ZKUtil(162): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,373 DEBUG [RS:0;jenkins-hbase4:39591] zookeeper.ZKUtil(162): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,373 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60805300, jitterRate=-0.09393042325973511}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-10 13:54:05,373 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676037245329Initializing all the Stores at 1676037245331 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676037245331Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037245331Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676037245331Cleaning up temporary data from old regions at 1676037245356 (+25 ms)Cleaning up detritus from prior splits at 1676037245357 (+1 ms)Region opened successfully at 1676037245373 (+16 ms) 2023-02-10 13:54:05,373 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-10 13:54:05,373 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-10 13:54:05,374 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-10 13:54:05,374 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-10 13:54:05,374 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-10 13:54:05,376 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-10 13:54:05,377 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676037245373Disabling compacts and flushes for region at 1676037245373Disabling writes for close at 1676037245374 (+1 ms)Writing region close event to WAL at 1676037245376 (+2 ms)Closed at 1676037245376 2023-02-10 13:54:05,383 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:05,383 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-10 13:54:05,386 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-10 13:54:05,386 DEBUG [RS:1;jenkins-hbase4:46307] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-10 13:54:05,386 DEBUG [RS:2;jenkins-hbase4:40651] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-10 13:54:05,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-10 13:54:05,397 INFO [RS:1;jenkins-hbase4:46307] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-10 13:54:05,397 INFO [RS:2;jenkins-hbase4:40651] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-10 13:54:05,397 INFO [RS:0;jenkins-hbase4:39591] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-10 13:54:05,417 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-10 13:54:05,419 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-10 13:54:05,424 INFO [RS:1;jenkins-hbase4:46307] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-10 13:54:05,424 INFO [RS:2;jenkins-hbase4:40651] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-10 13:54:05,424 INFO [RS:0;jenkins-hbase4:39591] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-10 13:54:05,429 INFO [RS:0;jenkins-hbase4:39591] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-10 13:54:05,429 INFO [RS:1;jenkins-hbase4:46307] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-10 13:54:05,430 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,429 INFO [RS:2;jenkins-hbase4:40651] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-10 13:54:05,430 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,430 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,433 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-10 13:54:05,433 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-10 13:54:05,433 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-10 13:54:05,441 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,441 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,441 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,441 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,441 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,441 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,441 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,441 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,441 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-10 13:54:05,442 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-10 13:54:05,443 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:1;jenkins-hbase4:46307] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,442 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:2;jenkins-hbase4:40651] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,443 DEBUG [RS:0;jenkins-hbase4:39591] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-10 13:54:05,448 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,448 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,448 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,448 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,448 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,448 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,448 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,449 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,449 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,467 INFO [RS:2;jenkins-hbase4:40651] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-10 13:54:05,467 INFO [RS:1;jenkins-hbase4:46307] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-10 13:54:05,467 INFO [RS:0;jenkins-hbase4:39591] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-10 13:54:05,470 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40651,1676037244010-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,470 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39591,1676037243910-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,470 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46307,1676037243975-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:05,486 INFO [RS:1;jenkins-hbase4:46307] regionserver.Replication(203): jenkins-hbase4.apache.org,46307,1676037243975 started 2023-02-10 13:54:05,486 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46307,1676037243975, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46307, sessionid=0x1003e82a6100002 2023-02-10 13:54:05,486 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-10 13:54:05,486 DEBUG [RS:1;jenkins-hbase4:46307] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,486 DEBUG [RS:1;jenkins-hbase4:46307] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46307,1676037243975' 2023-02-10 13:54:05,486 DEBUG [RS:1;jenkins-hbase4:46307] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:05,487 DEBUG [RS:1;jenkins-hbase4:46307] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:05,487 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-10 13:54:05,487 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-10 13:54:05,487 DEBUG [RS:1;jenkins-hbase4:46307] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:05,487 DEBUG [RS:1;jenkins-hbase4:46307] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46307,1676037243975' 2023-02-10 13:54:05,487 DEBUG [RS:1;jenkins-hbase4:46307] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-10 13:54:05,488 DEBUG [RS:1;jenkins-hbase4:46307] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-10 13:54:05,488 DEBUG [RS:1;jenkins-hbase4:46307] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-10 13:54:05,488 INFO [RS:1;jenkins-hbase4:46307] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-10 13:54:05,488 INFO [RS:1;jenkins-hbase4:46307] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-10 13:54:05,490 INFO [RS:0;jenkins-hbase4:39591] regionserver.Replication(203): jenkins-hbase4.apache.org,39591,1676037243910 started 2023-02-10 13:54:05,490 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39591,1676037243910, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39591, sessionid=0x1003e82a6100001 2023-02-10 13:54:05,492 INFO [RS:2;jenkins-hbase4:40651] regionserver.Replication(203): jenkins-hbase4.apache.org,40651,1676037244010 started 2023-02-10 13:54:05,492 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-10 13:54:05,492 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40651,1676037244010, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40651, sessionid=0x1003e82a6100003 2023-02-10 13:54:05,492 DEBUG [RS:0;jenkins-hbase4:39591] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,493 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-10 13:54:05,493 DEBUG [RS:2;jenkins-hbase4:40651] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,493 DEBUG [RS:2;jenkins-hbase4:40651] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40651,1676037244010' 2023-02-10 13:54:05,493 DEBUG [RS:0;jenkins-hbase4:39591] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39591,1676037243910' 2023-02-10 13:54:05,493 DEBUG [RS:0;jenkins-hbase4:39591] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:05,493 DEBUG [RS:2;jenkins-hbase4:40651] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:05,494 DEBUG [RS:0;jenkins-hbase4:39591] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:05,494 DEBUG [RS:2;jenkins-hbase4:40651] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:05,494 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-10 13:54:05,494 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-10 13:54:05,494 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-10 13:54:05,494 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-10 13:54:05,495 DEBUG [RS:2;jenkins-hbase4:40651] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:05,495 DEBUG [RS:2;jenkins-hbase4:40651] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40651,1676037244010' 2023-02-10 13:54:05,495 DEBUG [RS:2;jenkins-hbase4:40651] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-10 13:54:05,495 DEBUG [RS:0;jenkins-hbase4:39591] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,495 DEBUG [RS:0;jenkins-hbase4:39591] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39591,1676037243910' 2023-02-10 13:54:05,495 DEBUG [RS:0;jenkins-hbase4:39591] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-10 13:54:05,496 DEBUG [RS:2;jenkins-hbase4:40651] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-10 13:54:05,496 DEBUG [RS:0;jenkins-hbase4:39591] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-10 13:54:05,496 DEBUG [RS:2;jenkins-hbase4:40651] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-10 13:54:05,496 INFO [RS:2;jenkins-hbase4:40651] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-10 13:54:05,497 INFO [RS:2;jenkins-hbase4:40651] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-10 13:54:05,497 DEBUG [RS:0;jenkins-hbase4:39591] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-10 13:54:05,497 INFO [RS:0;jenkins-hbase4:39591] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-10 13:54:05,497 INFO [RS:0;jenkins-hbase4:39591] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-10 13:54:05,571 DEBUG [jenkins-hbase4:38205] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-10 13:54:05,575 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:05,581 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:05,581 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:05,581 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:05,581 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:05,584 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39591,1676037243910, state=OPENING 2023-02-10 13:54:05,592 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-10 13:54:05,593 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:05,594 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-10 13:54:05,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39591,1676037243910}] 2023-02-10 13:54:05,601 INFO [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39591%2C1676037243910, suffix=, logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,39591,1676037243910, archiveDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs, maxLogs=32 2023-02-10 13:54:05,601 INFO [RS:2;jenkins-hbase4:40651] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40651%2C1676037244010, suffix=, logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,40651,1676037244010, archiveDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs, maxLogs=32 2023-02-10 13:54:05,602 INFO [RS:1;jenkins-hbase4:46307] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46307%2C1676037243975, suffix=, logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,46307,1676037243975, archiveDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs, maxLogs=32 2023-02-10 13:54:05,630 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK] 2023-02-10 13:54:05,630 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK] 2023-02-10 13:54:05,631 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK] 2023-02-10 13:54:05,639 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK] 2023-02-10 13:54:05,640 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK] 2023-02-10 13:54:05,640 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK] 2023-02-10 13:54:05,645 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK] 2023-02-10 13:54:05,645 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK] 2023-02-10 13:54:05,646 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK] 2023-02-10 13:54:05,658 INFO [RS:1;jenkins-hbase4:46307] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,46307,1676037243975/jenkins-hbase4.apache.org%2C46307%2C1676037243975.1676037245607 2023-02-10 13:54:05,659 INFO [RS:2;jenkins-hbase4:40651] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,40651,1676037244010/jenkins-hbase4.apache.org%2C40651%2C1676037244010.1676037245607 2023-02-10 13:54:05,659 INFO [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,39591,1676037243910/jenkins-hbase4.apache.org%2C39591%2C1676037243910.1676037245607 2023-02-10 13:54:05,662 DEBUG [RS:1;jenkins-hbase4:46307] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK], DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK]] 2023-02-10 13:54:05,662 DEBUG [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK], DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK], DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK]] 2023-02-10 13:54:05,663 DEBUG [RS:2;jenkins-hbase4:40651] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK], DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK]] 2023-02-10 13:54:05,786 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:05,790 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-10 13:54:05,793 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35982, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-10 13:54:05,805 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-10 13:54:05,805 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-10 13:54:05,809 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39591%2C1676037243910.meta, suffix=.meta, logDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,39591,1676037243910, archiveDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs, maxLogs=32 2023-02-10 13:54:05,831 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK] 2023-02-10 13:54:05,831 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK] 2023-02-10 13:54:05,834 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK] 2023-02-10 13:54:05,840 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,39591,1676037243910/jenkins-hbase4.apache.org%2C39591%2C1676037243910.meta.1676037245811.meta 2023-02-10 13:54:05,840 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40385,DS-3ac8c082-b9e4-4c26-9163-07255a9eb98f,DISK], DatanodeInfoWithStorage[127.0.0.1:39211,DS-5f6d3772-211e-40b9-9dba-c822c7c2d7e8,DISK], DatanodeInfoWithStorage[127.0.0.1:36191,DS-17591b7b-de3e-43bb-863b-f2e0e725e205,DISK]] 2023-02-10 13:54:05,840 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:05,845 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:05,845 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-10 13:54:05,862 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-10 13:54:05,865 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-10 13:54:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-10 13:54:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-10 13:54:05,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-10 13:54:05,872 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-10 13:54:05,874 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/info 2023-02-10 13:54:05,874 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/info 2023-02-10 13:54:05,875 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-10 13:54:05,876 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:05,876 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-10 13:54:05,877 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/rep_barrier 2023-02-10 13:54:05,877 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/rep_barrier 2023-02-10 13:54:05,878 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-10 13:54:05,879 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:05,879 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-10 13:54:05,880 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/table 2023-02-10 13:54:05,880 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/table 2023-02-10 13:54:05,881 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-10 13:54:05,881 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:05,884 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740 2023-02-10 13:54:05,887 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740 2023-02-10 13:54:05,892 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-10 13:54:05,895 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-10 13:54:05,897 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59921922, jitterRate=-0.10709378123283386}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-10 13:54:05,897 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676037245870Writing region info on filesystem at 1676037245870Initializing all the Stores at 1676037245872 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676037245872Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037245872Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676037245872Cleaning up temporary data from old regions at 1676037245889 (+17 ms)Cleaning up detritus from prior splits at 1676037245891 (+2 ms)Running coprocessor post-open hooks at 1676037245897 (+6 ms)Region opened successfully at 1676037245897 2023-02-10 13:54:05,907 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676037245777 2023-02-10 13:54:05,924 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-10 13:54:05,925 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-10 13:54:05,925 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39591,1676037243910, state=OPEN 2023-02-10 13:54:05,928 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-10 13:54:05,928 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-10 13:54:05,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-10 13:54:05,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39591,1676037243910 in 330 msec 2023-02-10 13:54:05,937 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-10 13:54:05,937 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 540 msec 2023-02-10 13:54:05,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 946 msec 2023-02-10 13:54:05,943 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676037244052, completionTime=-1 2023-02-10 13:54:05,943 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-10 13:54:05,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:06,011 DEBUG [hconnection-0x28c66558-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-10 13:54:06,014 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35992, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-10 13:54:06,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:06,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037306030 2023-02-10 13:54:06,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037366030 2023-02-10 13:54:06,031 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 87 msec 2023-02-10 13:54:06,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38205,1676037242772-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:06,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38205,1676037242772-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:06,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38205,1676037242772-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:06,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38205, period=300000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:06,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-10 13:54:06,065 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-10 13:54:06,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-10 13:54:06,075 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-10 13:54:06,084 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:06,087 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:06,089 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:06,115 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,118 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1 empty. 2023-02-10 13:54:06,119 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,119 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-10 13:54:06,190 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:06,192 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 68b3720325c3cd17f2ad6817c681aaf1, NAME => 'hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:06,211 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:06,211 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 68b3720325c3cd17f2ad6817c681aaf1, disabling compactions & flushes 2023-02-10 13:54:06,211 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,211 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,212 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. after waiting 0 ms 2023-02-10 13:54:06,212 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,212 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,212 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 68b3720325c3cd17f2ad6817c681aaf1: Waiting for close lock at 1676037246211Disabling compacts and flushes for region at 1676037246211Disabling writes for close at 1676037246212 (+1 ms)Writing region close event to WAL at 1676037246212Closed at 1676037246212 2023-02-10 13:54:06,217 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:06,231 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676037246220"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037246220"}]},"ts":"1676037246220"} 2023-02-10 13:54:06,256 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:06,258 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:06,262 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037246258"}]},"ts":"1676037246258"} 2023-02-10 13:54:06,266 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-10 13:54:06,270 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:06,271 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:06,271 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:06,271 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:06,271 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:06,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=68b3720325c3cd17f2ad6817c681aaf1, ASSIGN}] 2023-02-10 13:54:06,278 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=68b3720325c3cd17f2ad6817c681aaf1, ASSIGN 2023-02-10 13:54:06,280 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=68b3720325c3cd17f2ad6817c681aaf1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39591,1676037243910; forceNewPlan=false, retain=false 2023-02-10 13:54:06,432 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:06,433 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:06,434 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676037246433"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037246433"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037246433"}]},"ts":"1676037246433"} 2023-02-10 13:54:06,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 68b3720325c3cd17f2ad6817c681aaf1, server=jenkins-hbase4.apache.org,39591,1676037243910}] 2023-02-10 13:54:06,600 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,602 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 68b3720325c3cd17f2ad6817c681aaf1, NAME => 'hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:06,603 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:06,603 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,603 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:06,603 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,603 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,605 INFO [StoreOpener-68b3720325c3cd17f2ad6817c681aaf1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,607 DEBUG [StoreOpener-68b3720325c3cd17f2ad6817c681aaf1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/info 2023-02-10 13:54:06,608 DEBUG [StoreOpener-68b3720325c3cd17f2ad6817c681aaf1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/info 2023-02-10 13:54:06,608 INFO [StoreOpener-68b3720325c3cd17f2ad6817c681aaf1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68b3720325c3cd17f2ad6817c681aaf1 columnFamilyName info 2023-02-10 13:54:06,609 INFO [StoreOpener-68b3720325c3cd17f2ad6817c681aaf1-1] regionserver.HStore(310): Store=68b3720325c3cd17f2ad6817c681aaf1/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:06,611 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,612 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,616 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:54:06,619 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:06,620 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 68b3720325c3cd17f2ad6817c681aaf1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68156617, jitterRate=0.015612736344337463}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-10 13:54:06,620 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 68b3720325c3cd17f2ad6817c681aaf1: Running coprocessor pre-open hook at 1676037246604Writing region info on filesystem at 1676037246604Initializing all the Stores at 1676037246605 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676037246605Cleaning up temporary data from old regions at 1676037246613 (+8 ms)Cleaning up detritus from prior splits at 1676037246613Running coprocessor post-open hooks at 1676037246620 (+7 ms)Region opened successfully at 1676037246620 2023-02-10 13:54:06,622 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1., pid=6, masterSystemTime=1676037246592 2023-02-10 13:54:06,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,626 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:54:06,627 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:06,628 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676037246626"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037246626"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037246626"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037246626"}]},"ts":"1676037246626"} 2023-02-10 13:54:06,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-10 13:54:06,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 68b3720325c3cd17f2ad6817c681aaf1, server=jenkins-hbase4.apache.org,39591,1676037243910 in 194 msec 2023-02-10 13:54:06,640 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-10 13:54:06,640 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=68b3720325c3cd17f2ad6817c681aaf1, ASSIGN in 362 msec 2023-02-10 13:54:06,641 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:06,642 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037246642"}]},"ts":"1676037246642"} 2023-02-10 13:54:06,644 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-10 13:54:06,647 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:06,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 571 msec 2023-02-10 13:54:06,687 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-10 13:54:06,690 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-10 13:54:06,690 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:06,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:06,735 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-10 13:54:06,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 26 msec 2023-02-10 13:54:06,743 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:06,754 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-10 13:54:06,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-02-10 13:54:06,769 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-10 13:54:06,772 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-10 13:54:06,772 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.706sec 2023-02-10 13:54:06,775 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-10 13:54:06,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-10 13:54:06,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-10 13:54:06,777 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38205,1676037242772-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-10 13:54:06,779 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38205,1676037242772-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-10 13:54:06,787 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-10 13:54:06,870 DEBUG [Listener at localhost/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x3ff4e667 to 127.0.0.1:63999 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-10 13:54:06,875 DEBUG [Listener at localhost/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c2173fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-10 13:54:06,889 DEBUG [hconnection-0x4b4df50e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-10 13:54:06,900 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36002, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-10 13:54:06,909 INFO [Listener at localhost/43477] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:54:06,938 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=358, OpenFileDescriptor=688, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=168, AvailableMemoryMB=3731 2023-02-10 13:54:06,942 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:06,950 DEBUG [Listener at localhost/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-10 13:54:06,953 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53780, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-10 13:54:06,966 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:54:06,966 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:54:06,967 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:54:06,972 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:06,974 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:06,976 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:06,979 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 2023-02-10 13:54:06,980 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 empty. 2023-02-10 13:54:06,983 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 2023-02-10 13:54:06,983 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-02-10 13:54:07,003 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:07,005 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(7671): creating {ENCODED => 7a96639de8473c08917073275e5ad055, NAME => 'testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionUnevenDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:07,028 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:07,029 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1603): Closing 7a96639de8473c08917073275e5ad055, disabling compactions & flushes 2023-02-10 13:54:07,029 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,029 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,029 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. after waiting 0 ms 2023-02-10 13:54:07,029 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,029 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,029 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1557): Region close journal for 7a96639de8473c08917073275e5ad055: Waiting for close lock at 1676037247029Disabling compacts and flushes for region at 1676037247029Disabling writes for close at 1676037247029Writing region close event to WAL at 1676037247029Closed at 1676037247029 2023-02-10 13:54:07,033 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:07,035 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676037247034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037247034"}]},"ts":"1676037247034"} 2023-02-10 13:54:07,037 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:07,038 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:07,038 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037247038"}]},"ts":"1676037247038"} 2023-02-10 13:54:07,040 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLING in hbase:meta 2023-02-10 13:54:07,045 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:07,046 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:07,046 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:07,046 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:07,046 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:07,047 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, ASSIGN}] 2023-02-10 13:54:07,049 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, ASSIGN 2023-02-10 13:54:07,050 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:54:07,201 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:07,202 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7a96639de8473c08917073275e5ad055, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,202 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676037247202"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037247202"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037247202"}]},"ts":"1676037247202"} 2023-02-10 13:54:07,205 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 7a96639de8473c08917073275e5ad055, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:07,359 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,359 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-10 13:54:07,363 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54860, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-10 13:54:07,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7a96639de8473c08917073275e5ad055, NAME => 'testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:07,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:07,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:07,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,374 INFO [StoreOpener-7a96639de8473c08917073275e5ad055-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,379 DEBUG [StoreOpener-7a96639de8473c08917073275e5ad055-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1 2023-02-10 13:54:07,379 DEBUG [StoreOpener-7a96639de8473c08917073275e5ad055-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1 2023-02-10 13:54:07,380 INFO [StoreOpener-7a96639de8473c08917073275e5ad055-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a96639de8473c08917073275e5ad055 columnFamilyName cf1 2023-02-10 13:54:07,381 INFO [StoreOpener-7a96639de8473c08917073275e5ad055-1] regionserver.HStore(310): Store=7a96639de8473c08917073275e5ad055/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:07,381 INFO [StoreOpener-7a96639de8473c08917073275e5ad055-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,384 DEBUG [StoreOpener-7a96639de8473c08917073275e5ad055-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2 2023-02-10 13:54:07,384 DEBUG [StoreOpener-7a96639de8473c08917073275e5ad055-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2 2023-02-10 13:54:07,384 INFO [StoreOpener-7a96639de8473c08917073275e5ad055-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a96639de8473c08917073275e5ad055 columnFamilyName cf2 2023-02-10 13:54:07,385 INFO [StoreOpener-7a96639de8473c08917073275e5ad055-1] regionserver.HStore(310): Store=7a96639de8473c08917073275e5ad055/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:07,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:07,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:07,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:07,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7a96639de8473c08917073275e5ad055; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70191726, jitterRate=0.04593822360038757}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:07,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7a96639de8473c08917073275e5ad055: Running coprocessor pre-open hook at 1676037247370Writing region info on filesystem at 1676037247370Initializing all the Stores at 1676037247372 (+2 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037247372Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037247374 (+2 ms)Cleaning up temporary data from old regions at 1676037247390 (+16 ms)Cleaning up detritus from prior splits at 1676037247391 (+1 ms)Running coprocessor post-open hooks at 1676037247405 (+14 ms)Region opened successfully at 1676037247405 2023-02-10 13:54:07,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055., pid=11, masterSystemTime=1676037247359 2023-02-10 13:54:07,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,415 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7a96639de8473c08917073275e5ad055, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,415 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676037247414"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037247414"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037247414"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037247414"}]},"ts":"1676037247414"} 2023-02-10 13:54:07,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-10 13:54:07,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 7a96639de8473c08917073275e5ad055, server=jenkins-hbase4.apache.org,40651,1676037244010 in 216 msec 2023-02-10 13:54:07,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-10 13:54:07,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, ASSIGN in 378 msec 2023-02-10 13:54:07,432 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:07,432 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037247432"}]},"ts":"1676037247432"} 2023-02-10 13:54:07,434 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLED in hbase:meta 2023-02-10 13:54:07,438 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:07,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter in 468 msec 2023-02-10 13:54:07,478 DEBUG [Listener at localhost/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-10 13:54:07,483 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54866, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-10 13:54:07,502 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:07,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,509 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:07,509 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,509 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-10 13:54:07,509 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-10 13:54:07,510 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,510 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,511 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,511 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,511 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,511 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,512 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,512 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:07,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,515 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:07,515 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:07,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:07,518 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,518 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. started... 2023-02-10 13:54:07,519 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:54:07,520 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,521 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,521 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,521 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:07,521 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,521 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:07,590 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/bccdb3afa1d340cd837935a2b0d61ac6 2023-02-10 13:54:07,650 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/272da14c24ca4b0480411d778f278f03 2023-02-10 13:54:07,660 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/bccdb3afa1d340cd837935a2b0d61ac6 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6 2023-02-10 13:54:07,670 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:07,673 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/272da14c24ca4b0480411d778f278f03 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03 2023-02-10 13:54:07,681 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:07,683 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 164ms, sequenceid=6, compaction requested=false 2023-02-10 13:54:07,684 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:07,686 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 7a96639de8473c08917073275e5ad055: 2023-02-10 13:54:07,686 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,686 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:07,687 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:07,687 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,687 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:07,687 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:07,689 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,689 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,690 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:07,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-10 13:54:07,693 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:07,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad77de9[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:07,693 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,698 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,699 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:07,699 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:07,700 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:07,703 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:07,703 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,703 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:07,703 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,704 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:07,710 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-10 13:54:07,710 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:07,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:07,710 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:07,710 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,710 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:07,712 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,712 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:07,712 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,713 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,713 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:07,714 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,714 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:07,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,717 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,717 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:07,718 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,731 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,731 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:07,731 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:07,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,731 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,734 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:07,734 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:07,845 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:07,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:07,880 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:07,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:07,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,883 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:07,883 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,884 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-10 13:54:07,884 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-10 13:54:07,885 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,885 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,886 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,886 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,886 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:07,886 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:07,887 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,887 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:07,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:07,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:07,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:07,889 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:07,891 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:07,891 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. started... 2023-02-10 13:54:07,892 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:07,893 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:07,914 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/0257b518e79b42e5ab6097f2f119d7e8 2023-02-10 13:54:07,940 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/57d5cb8466e949079c43107ad34eb86d 2023-02-10 13:54:07,950 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/0257b518e79b42e5ab6097f2f119d7e8 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8 2023-02-10 13:54:07,958 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:07,959 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/57d5cb8466e949079c43107ad34eb86d as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d 2023-02-10 13:54:07,969 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:07,970 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 78ms, sequenceid=19, compaction requested=false 2023-02-10 13:54:07,971 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 7a96639de8473c08917073275e5ad055: 2023-02-10 13:54:07,971 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:07,974 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,974 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:07,975 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,975 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:07,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-10 13:54:07,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1afcb3df[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:07,977 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:07,977 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:07,979 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,979 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:07,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:07,979 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:07,981 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:07,981 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,981 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:07,982 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:07,982 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:07,982 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,982 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:07,982 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:07,982 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:07,982 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:07,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:07,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:07,987 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-10 13:54:07,988 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:07,988 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:07,988 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,988 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:07,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,992 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:07,993 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,994 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:07,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:07,996 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:07,997 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:07,997 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:07,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:07,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,006 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,007 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,006 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,006 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:08,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,108 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:08,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:08,132 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:08,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:08,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,134 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,134 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,135 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-10 13:54:08,135 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-10 13:54:08,135 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,135 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,137 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,137 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,137 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,137 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,137 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,137 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:08,143 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,143 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. started... 2023-02-10 13:54:08,143 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:08,144 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,144 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,146 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,147 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,147 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,166 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/4f2f24eac2374f2582575fac112246f3 2023-02-10 13:54:08,191 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/5fa9c12d53314669911c24e1f0437e94 2023-02-10 13:54:08,201 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/4f2f24eac2374f2582575fac112246f3 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3 2023-02-10 13:54:08,208 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:08,209 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/5fa9c12d53314669911c24e1f0437e94 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94 2023-02-10 13:54:08,216 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:08,218 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 75ms, sequenceid=32, compaction requested=true 2023-02-10 13:54:08,218 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 7a96639de8473c08917073275e5ad055: 2023-02-10 13:54:08,218 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,218 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,221 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,221 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-10 13:54:08,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff9ae43[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:08,223 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:08,223 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,224 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,225 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:08,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:08,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:08,234 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-10 13:54:08,234 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:08,234 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:08,234 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,234 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,237 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,237 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,238 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,240 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,240 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,241 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,244 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,244 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,245 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:08,244 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,245 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,246 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:08,246 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:08,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,346 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:08,369 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:08,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:08,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,372 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,372 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,373 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-10 13:54:08,373 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-10 13:54:08,374 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,374 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,375 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,375 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,375 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,375 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,376 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,376 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,380 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,380 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,381 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,381 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,381 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,380 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,382 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,382 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. started... 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,382 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:08,383 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,385 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,386 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,386 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,386 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,386 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,386 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,406 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/ec78edd967164f8587f5ca488c328845 2023-02-10 13:54:08,438 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/86d718f57c854011b28025faf412d8c3 2023-02-10 13:54:08,446 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/ec78edd967164f8587f5ca488c328845 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845 2023-02-10 13:54:08,453 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:08,454 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/86d718f57c854011b28025faf412d8c3 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3 2023-02-10 13:54:08,462 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:08,463 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 81ms, sequenceid=45, compaction requested=true 2023-02-10 13:54:08,463 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 7a96639de8473c08917073275e5ad055: 2023-02-10 13:54:08,463 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,463 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,464 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,468 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-10 13:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3efcac35[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:08,470 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:08,470 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,472 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,472 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:08,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,473 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,475 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:08,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:08,479 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-10 13:54:08,479 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:08,479 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:08,479 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,479 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,482 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,481 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,482 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,483 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,485 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,486 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,486 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,490 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,489 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,489 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,490 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:08,489 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,490 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,491 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,591 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:08,614 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:08,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:08,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,616 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,616 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,617 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-10 13:54:08,617 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-10 13:54:08,618 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,618 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,619 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,619 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,619 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,619 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,619 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,619 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,621 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,621 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,621 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,621 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,622 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,623 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,624 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:08,625 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:08,625 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,625 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. started... 2023-02-10 13:54:08,625 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:08,627 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,628 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,641 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 2023-02-10 13:54:08,666 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/ba4261e37ae94f72a54964ab636d7935 2023-02-10 13:54:08,674 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 2023-02-10 13:54:08,681 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:08,682 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/ba4261e37ae94f72a54964ab636d7935 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935 2023-02-10 13:54:08,689 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:08,690 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 65ms, sequenceid=58, compaction requested=true 2023-02-10 13:54:08,690 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 7a96639de8473c08917073275e5ad055: 2023-02-10 13:54:08,690 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,690 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,693 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,693 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,693 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,693 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-10 13:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f25b636[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:08,695 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:08,696 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,697 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:08,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:08,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,697 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,697 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,697 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:08,698 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,698 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,700 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,701 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,701 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,701 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,702 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,702 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,702 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:08,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:08,705 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-10 13:54:08,706 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:08,706 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:08,706 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,706 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,708 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,708 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,709 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,710 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,711 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,711 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,717 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,717 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,718 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:08,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,717 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,718 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,819 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:08,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:08,843 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:08,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,846 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,846 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,847 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-10 13:54:08,847 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-10 13:54:08,847 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,847 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,848 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,848 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,848 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,848 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-10 13:54:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-10 13:54:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,851 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,853 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-10 13:54:08,853 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,852 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,853 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,853 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:08,853 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:08,853 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,853 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,853 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. started... 2023-02-10 13:54:08,853 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:08,854 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,856 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,873 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/c48653d70b10443bb72d85fc04f5f07f 2023-02-10 13:54:08,893 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/795077f94b8840228fcf01c0faa65215 2023-02-10 13:54:08,900 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/c48653d70b10443bb72d85fc04f5f07f as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f 2023-02-10 13:54:08,906 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:08,907 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/795077f94b8840228fcf01c0faa65215 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215 2023-02-10 13:54:08,915 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:08,916 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 63ms, sequenceid=71, compaction requested=true 2023-02-10 13:54:08,916 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 7a96639de8473c08917073275e5ad055: 2023-02-10 13:54:08,916 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:08,917 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:08,917 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:08,917 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,917 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-10 13:54:08,917 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-10 13:54:08,919 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,919 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:08,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-10 13:54:08,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20389446[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:08,922 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:08,922 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,923 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-10 13:54:08,924 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:08,923 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:08,923 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,925 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,926 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-10 13:54:08,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,927 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-10 13:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:08,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:08,931 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-10 13:54:08,932 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:08,932 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:08,932 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,932 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:08,933 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:08,933 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,933 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,933 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:08,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,934 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,934 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,934 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:08,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:08,937 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,937 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,937 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,940 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,940 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-10 13:54:08,941 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:08,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,940 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:08,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:08,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:08,941 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,942 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:08,942 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:08,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:08,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,942 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:08,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:08,942 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:08,942 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-10 13:54:09,042 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:09,064 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:09,070 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:09,071 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:09,071 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:09,071 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:09,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, UNASSIGN}] 2023-02-10 13:54:09,080 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, UNASSIGN 2023-02-10 13:54:09,081 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7a96639de8473c08917073275e5ad055, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:09,081 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676037249081"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037249081"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037249081"}]},"ts":"1676037249081"} 2023-02-10 13:54:09,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 7a96639de8473c08917073275e5ad055, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:09,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7a96639de8473c08917073275e5ad055, disabling compactions & flushes 2023-02-10 13:54:09,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:09,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:09,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. after waiting 0 ms 2023-02-10 13:54:09,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:09,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7a96639de8473c08917073275e5ad055 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:54:09,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/76a7abcaa6fc432eb0ad28bfebada17d 2023-02-10 13:54:09,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/4c0212940f8f4b95accfadc6b240c6d1 2023-02-10 13:54:09,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf1/76a7abcaa6fc432eb0ad28bfebada17d as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d 2023-02-10 13:54:09,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:54:09,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.tmp/cf2/4c0212940f8f4b95accfadc6b240c6d1 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1 2023-02-10 13:54:09,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:54:09,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 72ms, sequenceid=82, compaction requested=true 2023-02-10 13:54:09,335 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:54:09,336 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:09,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. 2023-02-10 13:54:09,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7a96639de8473c08917073275e5ad055: Waiting for close lock at 1676037249242Running coprocessor pre-close hooks at 1676037249242Disabling compacts and flushes for region at 1676037249242Disabling writes for close at 1676037249243 (+1 ms)Obtaining lock to block concurrent updates at 1676037249243Preparing flush snapshotting stores in 7a96639de8473c08917073275e5ad055 at 1676037249243Finished memstore snapshotting testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1676037249243Flushing stores of testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. at 1676037249244 (+1 ms)Flushing 7a96639de8473c08917073275e5ad055/cf1: creating writer at 1676037249245 (+1 ms)Flushing 7a96639de8473c08917073275e5ad055/cf1: appending metadata at 1676037249248 (+3 ms)Flushing 7a96639de8473c08917073275e5ad055/cf1: closing flushed file at 1676037249248Flushing 7a96639de8473c08917073275e5ad055/cf2: creating writer at 1676037249273 (+25 ms)Flushing 7a96639de8473c08917073275e5ad055/cf2: appending metadata at 1676037249277 (+4 ms)Flushing 7a96639de8473c08917073275e5ad055/cf2: closing flushed file at 1676037249277Flushing 7a96639de8473c08917073275e5ad055/cf1: reopening flushed file at 1676037249299 (+22 ms)Flushing 7a96639de8473c08917073275e5ad055/cf2: reopening flushed file at 1676037249308 (+9 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 7a96639de8473c08917073275e5ad055 in 72ms, sequenceid=82, compaction requested=true at 1676037249315 (+7 ms)Writing region close event to WAL at 1676037249331 (+16 ms)Running coprocessor post-close hooks at 1676037249336 (+5 ms)Closed at 1676037249337 (+1 ms) 2023-02-10 13:54:09,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,340 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=7a96639de8473c08917073275e5ad055, regionState=CLOSED 2023-02-10 13:54:09,341 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676037249340"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037249340"}]},"ts":"1676037249340"} 2023-02-10 13:54:09,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-10 13:54:09,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 7a96639de8473c08917073275e5ad055, server=jenkins-hbase4.apache.org,40651,1676037244010 in 260 msec 2023-02-10 13:54:09,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-10 13:54:09,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=7a96639de8473c08917073275e5ad055, UNASSIGN in 266 msec 2023-02-10 13:54:09,367 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=12 splitting 14 storefiles, region=7a96639de8473c08917073275e5ad055, threads=14 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,368 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,369 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,369 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,369 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,369 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,370 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,370 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,372 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,406 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935, top=true 2023-02-10 13:54:09,414 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215, top=true 2023-02-10 13:54:09,438 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d, top=true 2023-02-10 13:54:09,441 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1, top=true 2023-02-10 13:54:09,447 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f, top=true 2023-02-10 13:54:09,462 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49, top=true 2023-02-10 13:54:09,500 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.splits/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215 for child: 47f7f8f648c73c9a6d503e945e957190, parent: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,500 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,506 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.splits/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f for child: 47f7f8f648c73c9a6d503e945e957190, parent: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,506 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,518 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.splits/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d for child: 47f7f8f648c73c9a6d503e945e957190, parent: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,518 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,518 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.splits/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935 for child: 47f7f8f648c73c9a6d503e945e957190, parent: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,518 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,520 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.splits/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1 for child: 47f7f8f648c73c9a6d503e945e957190, parent: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,520 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,533 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/.splits/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49 for child: 47f7f8f648c73c9a6d503e945e957190, parent: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,534 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,600 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,611 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,611 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,612 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,612 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,617 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,617 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,617 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3 for region: 7a96639de8473c08917073275e5ad055 2023-02-10 13:54:09,618 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=12 split storefiles for region 7a96639de8473c08917073275e5ad055 Daughter A: 8 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:54:09,670 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:54:09,672 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:54:09,675 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676037249675"},{"qualifier":"splitA","vlen":70,"tag":[],"timestamp":"1676037249675"},{"qualifier":"splitB","vlen":70,"tag":[],"timestamp":"1676037249675"}]},"ts":"1676037249675"} 2023-02-10 13:54:09,675 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037249675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037249675"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037249675"}]},"ts":"1676037249675"} 2023-02-10 13:54:09,675 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037249675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037249675"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037249675"}]},"ts":"1676037249675"} 2023-02-10 13:54:09,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, ASSIGN}] 2023-02-10 13:54:09,724 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, ASSIGN 2023-02-10 13:54:09,725 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:54:09,726 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, ASSIGN 2023-02-10 13:54:09,727 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:54:09,876 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:54:09,877 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=47f7f8f648c73c9a6d503e945e957190, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:09,878 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037249877"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037249877"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037249877"}]},"ts":"1676037249877"} 2023-02-10 13:54:09,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 47f7f8f648c73c9a6d503e945e957190, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:09,884 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=156bb80eea8edfca1cd7027bc93f038b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:09,884 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037249884"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037249884"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037249884"}]},"ts":"1676037249884"} 2023-02-10 13:54:09,886 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure 156bb80eea8edfca1cd7027bc93f038b, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:10,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 47f7f8f648c73c9a6d503e945e957190, NAME => 'testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.', STARTKEY => '26', ENDKEY => ''} 2023-02-10 13:54:10,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:10,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:10,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,039 INFO [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,040 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1 2023-02-10 13:54:10,040 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1 2023-02-10 13:54:10,041 INFO [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 47f7f8f648c73c9a6d503e945e957190 columnFamilyName cf1 2023-02-10 13:54:10,077 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8-top 2023-02-10 13:54:10,084 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3-top 2023-02-10 13:54:10,089 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6-top 2023-02-10 13:54:10,095 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845-top 2023-02-10 13:54:10,101 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d 2023-02-10 13:54:10,105 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49 2023-02-10 13:54:10,109 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f 2023-02-10 13:54:10,110 INFO [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(310): Store=47f7f8f648c73c9a6d503e945e957190/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:10,110 INFO [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,111 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2 2023-02-10 13:54:10,111 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2 2023-02-10 13:54:10,112 INFO [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 47f7f8f648c73c9a6d503e945e957190 columnFamilyName cf2 2023-02-10 13:54:10,125 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03-top 2023-02-10 13:54:10,130 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d-top 2023-02-10 13:54:10,135 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94-top 2023-02-10 13:54:10,141 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3-top 2023-02-10 13:54:10,146 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1 2023-02-10 13:54:10,150 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215 2023-02-10 13:54:10,155 DEBUG [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935 2023-02-10 13:54:10,155 INFO [StoreOpener-47f7f8f648c73c9a6d503e945e957190-1] regionserver.HStore(310): Store=47f7f8f648c73c9a6d503e945e957190/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:10,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:10,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 47f7f8f648c73c9a6d503e945e957190; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58935021, jitterRate=-0.12179975211620331}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:10,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 47f7f8f648c73c9a6d503e945e957190: Running coprocessor pre-open hook at 1676037250038Writing region info on filesystem at 1676037250038Initializing all the Stores at 1676037250039 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037250039Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037250039Cleaning up temporary data from old regions at 1676037250159 (+120 ms)Cleaning up detritus from prior splits at 1676037250159Running coprocessor post-open hooks at 1676037250162 (+3 ms)Region opened successfully at 1676037250162 2023-02-10 13:54:10,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190., pid=17, masterSystemTime=1676037250032 2023-02-10 13:54:10,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:10,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:10,167 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:10,167 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:10,170 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,170 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): 47f7f8f648c73c9a6d503e945e957190/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:10,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,170 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,170 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): 47f7f8f648c73c9a6d503e945e957190/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:10,170 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 47f7f8f648c73c9a6d503e945e957190/cf2 in testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,170 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 47f7f8f648c73c9a6d503e945e957190/cf1 in testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 156bb80eea8edfca1cd7027bc93f038b, NAME => 'testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.', STARTKEY => '', ENDKEY => '26'} 2023-02-10 13:54:10,170 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=47f7f8f648c73c9a6d503e945e957190, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:10,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:10,171 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/.tmp, totalSize=34.9 K 2023-02-10 13:54:10,171 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/.tmp, totalSize=34.9 K 2023-02-10 13:54:10,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,171 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037250170"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037250170"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037250170"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037250170"}]},"ts":"1676037250170"} 2023-02-10 13:54:10,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:10,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,174 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037247484 2023-02-10 13:54:10,174 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037247484 2023-02-10 13:54:10,176 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037247853 2023-02-10 13:54:10,176 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037247853 2023-02-10 13:54:10,176 INFO [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,177 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037248110 2023-02-10 13:54:10,177 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037248110 2023-02-10 13:54:10,178 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037248348 2023-02-10 13:54:10,178 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037248348 2023-02-10 13:54:10,179 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1 2023-02-10 13:54:10,179 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037248593 2023-02-10 13:54:10,179 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1 2023-02-10 13:54:10,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-10 13:54:10,179 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037248593 2023-02-10 13:54:10,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 47f7f8f648c73c9a6d503e945e957190, server=jenkins-hbase4.apache.org,40651,1676037244010 in 296 msec 2023-02-10 13:54:10,180 INFO [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 156bb80eea8edfca1cd7027bc93f038b columnFamilyName cf1 2023-02-10 13:54:10,180 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037248822 2023-02-10 13:54:10,180 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037248822 2023-02-10 13:54:10,182 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037249045 2023-02-10 13:54:10,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, ASSIGN in 456 msec 2023-02-10 13:54:10,183 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037249045 2023-02-10 13:54:10,202 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8-bottom 2023-02-10 13:54:10,208 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3-bottom 2023-02-10 13:54:10,219 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6-bottom 2023-02-10 13:54:10,220 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): 47f7f8f648c73c9a6d503e945e957190#cf2#compaction#14 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:10,226 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845-bottom 2023-02-10 13:54:10,228 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): 47f7f8f648c73c9a6d503e945e957190#cf1#compaction#15 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:10,229 INFO [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(310): Store=156bb80eea8edfca1cd7027bc93f038b/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:10,230 INFO [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,231 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2 2023-02-10 13:54:10,231 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2 2023-02-10 13:54:10,232 INFO [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 156bb80eea8edfca1cd7027bc93f038b columnFamilyName cf2 2023-02-10 13:54:10,247 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03-bottom 2023-02-10 13:54:10,253 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d-bottom 2023-02-10 13:54:10,259 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/.tmp/cf2/d949f296f2e74e0ca09328a7353e60c8 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/d949f296f2e74e0ca09328a7353e60c8 2023-02-10 13:54:10,262 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94-bottom 2023-02-10 13:54:10,271 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/.tmp/cf1/dd1c0587f8b141f799142339f3b42f15 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/dd1c0587f8b141f799142339f3b42f15 2023-02-10 13:54:10,274 DEBUG [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3-bottom 2023-02-10 13:54:10,274 INFO [StoreOpener-156bb80eea8edfca1cd7027bc93f038b-1] regionserver.HStore(310): Store=156bb80eea8edfca1cd7027bc93f038b/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:10,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:10,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 156bb80eea8edfca1cd7027bc93f038b; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63888386, jitterRate=-0.04798886179924011}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:10,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 156bb80eea8edfca1cd7027bc93f038b: Running coprocessor pre-open hook at 1676037250173Writing region info on filesystem at 1676037250173Initializing all the Stores at 1676037250175 (+2 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037250175Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037250176 (+1 ms)Cleaning up temporary data from old regions at 1676037250278 (+102 ms)Cleaning up detritus from prior splits at 1676037250278Running coprocessor post-open hooks at 1676037250283 (+5 ms)Region opened successfully at 1676037250283 2023-02-10 13:54:10,285 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 47f7f8f648c73c9a6d503e945e957190/cf2 of 47f7f8f648c73c9a6d503e945e957190 into d949f296f2e74e0ca09328a7353e60c8(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:10,285 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 47f7f8f648c73c9a6d503e945e957190: 2023-02-10 13:54:10,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b., pid=18, masterSystemTime=1676037250032 2023-02-10 13:54:10,285 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190., storeName=47f7f8f648c73c9a6d503e945e957190/cf2, priority=9, startTime=1676037250166; duration=0sec 2023-02-10 13:54:10,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:10,285 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 47f7f8f648c73c9a6d503e945e957190/cf1 of 47f7f8f648c73c9a6d503e945e957190 into dd1c0587f8b141f799142339f3b42f15(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:10,285 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 47f7f8f648c73c9a6d503e945e957190: 2023-02-10 13:54:10,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-02-10 13:54:10,285 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190., storeName=47f7f8f648c73c9a6d503e945e957190/cf1, priority=9, startTime=1676037250165; duration=0sec 2023-02-10 13:54:10,286 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-02-10 13:54:10,287 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-02-10 13:54:10,287 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,287 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): 156bb80eea8edfca1cd7027bc93f038b/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:10,287 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 156bb80eea8edfca1cd7027bc93f038b/cf1 in testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,288 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/.tmp, totalSize=19.9 K 2023-02-10 13:54:10,288 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:10,288 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-02-10 13:54:10,289 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,289 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): 156bb80eea8edfca1cd7027bc93f038b/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:10,289 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 156bb80eea8edfca1cd7027bc93f038b/cf2 in testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,289 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037247484 2023-02-10 13:54:10,289 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/.tmp, totalSize=19.9 K 2023-02-10 13:54:10,290 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037247853 2023-02-10 13:54:10,290 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037247484 2023-02-10 13:54:10,290 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037248110 2023-02-10 13:54:10,291 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037247853 2023-02-10 13:54:10,291 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037248348 2023-02-10 13:54:10,291 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037248110 2023-02-10 13:54:10,292 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037248348 2023-02-10 13:54:10,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,295 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=156bb80eea8edfca1cd7027bc93f038b, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:10,295 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037250295"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037250295"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037250295"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037250295"}]},"ts":"1676037250295"} 2023-02-10 13:54:10,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-02-10 13:54:10,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure 156bb80eea8edfca1cd7027bc93f038b, server=jenkins-hbase4.apache.org,40651,1676037244010 in 412 msec 2023-02-10 13:54:10,307 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-02-10 13:54:10,307 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, ASSIGN in 580 msec 2023-02-10 13:54:10,309 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): 156bb80eea8edfca1cd7027bc93f038b#cf1#compaction#16 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:10,310 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 in 1.2430 sec 2023-02-10 13:54:10,317 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): 156bb80eea8edfca1cd7027bc93f038b#cf2#compaction#17 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:10,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40651] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Scan size: 157 connection: 172.31.14.131:54866 deadline: 1676037310325, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055. is not online on jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:10,334 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/.tmp/cf1/235226eb03f84daaad24910ee388d986 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/235226eb03f84daaad24910ee388d986 2023-02-10 13:54:10,338 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/.tmp/cf2/2356dd74296141f8ab999cf6e5896f1b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/2356dd74296141f8ab999cf6e5896f1b 2023-02-10 13:54:10,346 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 156bb80eea8edfca1cd7027bc93f038b/cf2 of 156bb80eea8edfca1cd7027bc93f038b into 2356dd74296141f8ab999cf6e5896f1b(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:10,346 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 156bb80eea8edfca1cd7027bc93f038b: 2023-02-10 13:54:10,346 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 156bb80eea8edfca1cd7027bc93f038b/cf1 of 156bb80eea8edfca1cd7027bc93f038b into 235226eb03f84daaad24910ee388d986(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:10,346 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b., storeName=156bb80eea8edfca1cd7027bc93f038b/cf2, priority=12, startTime=1676037250285; duration=0sec 2023-02-10 13:54:10,346 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 156bb80eea8edfca1cd7027bc93f038b: 2023-02-10 13:54:10,346 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b., storeName=156bb80eea8edfca1cd7027bc93f038b/cf1, priority=12, startTime=1676037250285; duration=0sec 2023-02-10 13:54:10,346 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:10,346 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:10,463 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-02-10 13:54:10,474 INFO [Listener at localhost/43477] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-10 13:54:10,477 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-02-10 13:54:10,505 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:10,516 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,520 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,534 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037250534"}]},"ts":"1676037250534"} 2023-02-10 13:54:10,536 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLING in hbase:meta 2023-02-10 13:54:10,538 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionUnevenDaughter to state=DISABLING 2023-02-10 13:54:10,540 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, UNASSIGN}, {pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, UNASSIGN}] 2023-02-10 13:54:10,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-02-10 13:54:10,541 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, UNASSIGN 2023-02-10 13:54:10,542 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=47f7f8f648c73c9a6d503e945e957190, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:10,542 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037250542"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037250542"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037250542"}]},"ts":"1676037250542"} 2023-02-10 13:54:10,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 47f7f8f648c73c9a6d503e945e957190, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:10,548 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, UNASSIGN 2023-02-10 13:54:10,549 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=156bb80eea8edfca1cd7027bc93f038b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:10,549 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037250549"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037250549"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037250549"}]},"ts":"1676037250549"} 2023-02-10 13:54:10,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE; CloseRegionProcedure 156bb80eea8edfca1cd7027bc93f038b, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:10,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-02-10 13:54:10,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 47f7f8f648c73c9a6d503e945e957190, disabling compactions & flushes 2023-02-10 13:54:10,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. after waiting 0 ms 2023-02-10 13:54:10,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,708 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d] to archive 2023-02-10 13:54:10,709 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:10,715 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,716 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,718 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,719 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,721 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-9b1a75ce4e514f5ba37a55428f7b2a49 2023-02-10 13:54:10,723 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-c48653d70b10443bb72d85fc04f5f07f 2023-02-10 13:54:10,724 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-76a7abcaa6fc432eb0ad28bfebada17d 2023-02-10 13:54:10,743 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1] to archive 2023-02-10 13:54:10,744 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:10,745 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,747 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,748 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,749 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,751 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-ba4261e37ae94f72a54964ab636d7935 2023-02-10 13:54:10,752 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-795077f94b8840228fcf01c0faa65215 2023-02-10 13:54:10,753 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/testSplitTableRegionUnevenDaughter=7a96639de8473c08917073275e5ad055-4c0212940f8f4b95accfadc6b240c6d1 2023-02-10 13:54:10,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:54:10,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:10,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190. 2023-02-10 13:54:10,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 47f7f8f648c73c9a6d503e945e957190: Waiting for close lock at 1676037250697Running coprocessor pre-close hooks at 1676037250697Disabling compacts and flushes for region at 1676037250697Disabling writes for close at 1676037250697Writing region close event to WAL at 1676037250754 (+57 ms)Running coprocessor post-close hooks at 1676037250758 (+4 ms)Closed at 1676037250758 2023-02-10 13:54:10,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 156bb80eea8edfca1cd7027bc93f038b, disabling compactions & flushes 2023-02-10 13:54:10,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. after waiting 0 ms 2023-02-10 13:54:10,760 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,761 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=47f7f8f648c73c9a6d503e945e957190, regionState=CLOSED 2023-02-10 13:54:10,761 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037250761"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037250761"}]},"ts":"1676037250761"} 2023-02-10 13:54:10,761 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845-bottom] to archive 2023-02-10 13:54:10,762 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:10,765 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/bccdb3afa1d340cd837935a2b0d61ac6.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-10 13:54:10,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 47f7f8f648c73c9a6d503e945e957190, server=jenkins-hbase4.apache.org,40651,1676037244010 in 219 msec 2023-02-10 13:54:10,767 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/0257b518e79b42e5ab6097f2f119d7e8.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=47f7f8f648c73c9a6d503e945e957190, UNASSIGN in 227 msec 2023-02-10 13:54:10,769 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/4f2f24eac2374f2582575fac112246f3.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,770 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/ec78edd967164f8587f5ca488c328845.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,772 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3-bottom] to archive 2023-02-10 13:54:10,773 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:10,775 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/272da14c24ca4b0480411d778f278f03.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,776 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/57d5cb8466e949079c43107ad34eb86d.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,777 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/5fa9c12d53314669911c24e1f0437e94.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,779 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/86d718f57c854011b28025faf412d8c3.7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:54:10,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:10,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b. 2023-02-10 13:54:10,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 156bb80eea8edfca1cd7027bc93f038b: Waiting for close lock at 1676037250760Running coprocessor pre-close hooks at 1676037250760Disabling compacts and flushes for region at 1676037250760Disabling writes for close at 1676037250760Writing region close event to WAL at 1676037250780 (+20 ms)Running coprocessor post-close hooks at 1676037250784 (+4 ms)Closed at 1676037250784 2023-02-10 13:54:10,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,787 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=156bb80eea8edfca1cd7027bc93f038b, regionState=CLOSED 2023-02-10 13:54:10,787 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676037250787"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037250787"}]},"ts":"1676037250787"} 2023-02-10 13:54:10,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=20 2023-02-10 13:54:10,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=20, state=SUCCESS; CloseRegionProcedure 156bb80eea8edfca1cd7027bc93f038b, server=jenkins-hbase4.apache.org,40651,1676037244010 in 238 msec 2023-02-10 13:54:10,794 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-02-10 13:54:10,794 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=156bb80eea8edfca1cd7027bc93f038b, UNASSIGN in 252 msec 2023-02-10 13:54:10,795 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037250795"}]},"ts":"1676037250795"} 2023-02-10 13:54:10,797 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLED in hbase:meta 2023-02-10 13:54:10,799 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionUnevenDaughter to state=DISABLED 2023-02-10 13:54:10,801 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter in 278 msec 2023-02-10 13:54:10,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-02-10 13:54:10,848 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 19 completed 2023-02-10 13:54:10,853 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,861 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,863 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=24, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-02-10 13:54:10,871 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,871 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,872 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,876 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/recovered.edits] 2023-02-10 13:54:10,876 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/recovered.edits] 2023-02-10 13:54:10,876 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/recovered.edits] 2023-02-10 13:54:10,898 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/dd1c0587f8b141f799142339f3b42f15 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf1/dd1c0587f8b141f799142339f3b42f15 2023-02-10 13:54:10,898 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/235226eb03f84daaad24910ee388d986 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf1/235226eb03f84daaad24910ee388d986 2023-02-10 13:54:10,898 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/0257b518e79b42e5ab6097f2f119d7e8 2023-02-10 13:54:10,900 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/4f2f24eac2374f2582575fac112246f3 2023-02-10 13:54:10,901 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/d949f296f2e74e0ca09328a7353e60c8 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/cf2/d949f296f2e74e0ca09328a7353e60c8 2023-02-10 13:54:10,902 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/2356dd74296141f8ab999cf6e5896f1b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/cf2/2356dd74296141f8ab999cf6e5896f1b 2023-02-10 13:54:10,902 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/76a7abcaa6fc432eb0ad28bfebada17d 2023-02-10 13:54:10,903 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/9b1a75ce4e514f5ba37a55428f7b2a49 2023-02-10 13:54:10,905 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/bccdb3afa1d340cd837935a2b0d61ac6 2023-02-10 13:54:10,905 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190/recovered.edits/90.seqid 2023-02-10 13:54:10,906 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b/recovered.edits/90.seqid 2023-02-10 13:54:10,908 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:10,913 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/156bb80eea8edfca1cd7027bc93f038b 2023-02-10 13:54:10,913 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/c48653d70b10443bb72d85fc04f5f07f 2023-02-10 13:54:10,916 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf1/ec78edd967164f8587f5ca488c328845 2023-02-10 13:54:10,920 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/272da14c24ca4b0480411d778f278f03 2023-02-10 13:54:10,922 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/4c0212940f8f4b95accfadc6b240c6d1 2023-02-10 13:54:10,924 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/57d5cb8466e949079c43107ad34eb86d 2023-02-10 13:54:10,925 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/5fa9c12d53314669911c24e1f0437e94 2023-02-10 13:54:10,927 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/795077f94b8840228fcf01c0faa65215 2023-02-10 13:54:10,929 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/86d718f57c854011b28025faf412d8c3 2023-02-10 13:54:10,930 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/cf2/ba4261e37ae94f72a54964ab636d7935 2023-02-10 13:54:10,934 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/recovered.edits/85.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055/recovered.edits/85.seqid 2023-02-10 13:54:10,935 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionUnevenDaughter/7a96639de8473c08917073275e5ad055 2023-02-10 13:54:10,935 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-02-10 13:54:10,938 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=24, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:10,959 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionUnevenDaughter from hbase:meta 2023-02-10 13:54:10,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-02-10 13:54:11,001 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionUnevenDaughter' descriptor. 2023-02-10 13:54:11,003 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=24, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:11,003 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionUnevenDaughter' from region states. 2023-02-10 13:54:11,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037251003"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:11,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037251003"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:11,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037251003"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:11,006 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:54:11,006 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7a96639de8473c08917073275e5ad055, NAME => 'testSplitTableRegionUnevenDaughter,,1676037246970.7a96639de8473c08917073275e5ad055.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 156bb80eea8edfca1cd7027bc93f038b, NAME => 'testSplitTableRegionUnevenDaughter,,1676037249064.156bb80eea8edfca1cd7027bc93f038b.', STARTKEY => '', ENDKEY => '26'}, {ENCODED => 47f7f8f648c73c9a6d503e945e957190, NAME => 'testSplitTableRegionUnevenDaughter,26,1676037249064.47f7f8f648c73c9a6d503e945e957190.', STARTKEY => '26', ENDKEY => ''}] 2023-02-10 13:54:11,006 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionUnevenDaughter' as deleted. 2023-02-10 13:54:11,006 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037251006"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:11,008 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionUnevenDaughter state from META 2023-02-10 13:54:11,011 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=24, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:11,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter in 157 msec 2023-02-10 13:54:11,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-02-10 13:54:11,167 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 24 completed 2023-02-10 13:54:11,184 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=390 (was 358) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:49702 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:38012 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:37658 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:37998 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42338 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28c66558-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:40651-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:49500 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:49732 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42340 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:49686 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:49680 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:37834 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42100 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:49768 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:37732 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42176 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28c66558-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42044 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=833 (was 688) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 254), ProcessCount=168 (was 168), AvailableMemoryMB=3646 (was 3731) 2023-02-10 13:54:11,198 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=390, OpenFileDescriptor=833, MaxFileDescriptor=60000, SystemLoadAverage=250, ProcessCount=168, AvailableMemoryMB=3645 2023-02-10 13:54:11,198 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:11,204 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:54:11,204 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:54:11,206 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:11,208 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:11,209 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:11,211 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,212 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c empty. 2023-02-10 13:54:11,212 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,213 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-02-10 13:54:11,227 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:11,229 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(7671): creating {ENCODED => 763a2d94ec8c0a841c2f6b5578c7623c, NAME => 'testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitWithoutPONR', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:11,237 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:11,237 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1603): Closing 763a2d94ec8c0a841c2f6b5578c7623c, disabling compactions & flushes 2023-02-10 13:54:11,237 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,237 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,237 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. after waiting 0 ms 2023-02-10 13:54:11,237 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,237 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,237 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1557): Region close journal for 763a2d94ec8c0a841c2f6b5578c7623c: Waiting for close lock at 1676037251237Disabling compacts and flushes for region at 1676037251237Disabling writes for close at 1676037251237Writing region close event to WAL at 1676037251237Closed at 1676037251237 2023-02-10 13:54:11,240 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:11,241 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037251241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037251241"}]},"ts":"1676037251241"} 2023-02-10 13:54:11,244 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:11,245 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:11,245 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037251245"}]},"ts":"1676037251245"} 2023-02-10 13:54:11,247 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLING in hbase:meta 2023-02-10 13:54:11,251 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:11,252 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:11,252 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:11,252 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:11,253 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:11,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, ASSIGN}] 2023-02-10 13:54:11,255 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, ASSIGN 2023-02-10 13:54:11,256 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:11,406 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:11,408 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=763a2d94ec8c0a841c2f6b5578c7623c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,408 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037251407"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037251407"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037251407"}]},"ts":"1676037251407"} 2023-02-10 13:54:11,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:11,415 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:54:11,489 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-10 13:54:11,490 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-10 13:54:11,563 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,563 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-10 13:54:11,566 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41722, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-10 13:54:11,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 763a2d94ec8c0a841c2f6b5578c7623c, NAME => 'testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:11,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:11,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:11,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,573 INFO [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,574 DEBUG [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1 2023-02-10 13:54:11,574 DEBUG [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1 2023-02-10 13:54:11,575 INFO [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 763a2d94ec8c0a841c2f6b5578c7623c columnFamilyName cf1 2023-02-10 13:54:11,576 INFO [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] regionserver.HStore(310): Store=763a2d94ec8c0a841c2f6b5578c7623c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:11,576 INFO [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,578 DEBUG [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2 2023-02-10 13:54:11,578 DEBUG [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2 2023-02-10 13:54:11,579 INFO [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 763a2d94ec8c0a841c2f6b5578c7623c columnFamilyName cf2 2023-02-10 13:54:11,580 INFO [StoreOpener-763a2d94ec8c0a841c2f6b5578c7623c-1] regionserver.HStore(310): Store=763a2d94ec8c0a841c2f6b5578c7623c/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:11,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:11,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:11,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:11,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 763a2d94ec8c0a841c2f6b5578c7623c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65731922, jitterRate=-0.02051803469657898}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:11,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 763a2d94ec8c0a841c2f6b5578c7623c: Running coprocessor pre-open hook at 1676037251571Writing region info on filesystem at 1676037251571Initializing all the Stores at 1676037251572 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037251572Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037251572Cleaning up temporary data from old regions at 1676037251582 (+10 ms)Cleaning up detritus from prior splits at 1676037251583 (+1 ms)Running coprocessor post-open hooks at 1676037251589 (+6 ms)Region opened successfully at 1676037251589 2023-02-10 13:54:11,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c., pid=27, masterSystemTime=1676037251563 2023-02-10 13:54:11,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,596 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=763a2d94ec8c0a841c2f6b5578c7623c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,596 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037251595"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037251595"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037251595"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037251595"}]},"ts":"1676037251595"} 2023-02-10 13:54:11,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-02-10 13:54:11,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 in 188 msec 2023-02-10 13:54:11,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-02-10 13:54:11,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, ASSIGN in 348 msec 2023-02-10 13:54:11,604 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:11,605 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037251604"}]},"ts":"1676037251604"} 2023-02-10 13:54:11,606 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLED in hbase:meta 2023-02-10 13:54:11,609 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:11,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR in 404 msec 2023-02-10 13:54:11,710 DEBUG [Listener at localhost/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-10 13:54:11,714 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41738, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-10 13:54:11,723 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:11,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-10 13:54:11,725 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-10 13:54:11,725 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,726 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-10 13:54:11,726 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-10 13:54:11,727 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,727 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,728 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,728 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,728 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:11,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:11,730 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:11,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:11,732 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,732 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. started... 2023-02-10 13:54:11,731 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,732 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:11,733 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:11,733 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:11,733 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:54:11,732 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:11,733 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:11,733 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:11,733 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:11,735 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,735 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,736 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,736 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,738 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:11,748 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/957aee19684d45b6b016046978054d2b 2023-02-10 13:54:11,762 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/3b795b2576a2412cb61d6fd4eb168b5c 2023-02-10 13:54:11,768 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/957aee19684d45b6b016046978054d2b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b 2023-02-10 13:54:11,773 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:11,774 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/3b795b2576a2412cb61d6fd4eb168b5c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c 2023-02-10 13:54:11,779 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:11,780 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 47ms, sequenceid=6, compaction requested=false 2023-02-10 13:54:11,780 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitWithoutPONR' 2023-02-10 13:54:11,780 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 763a2d94ec8c0a841c2f6b5578c7623c: 2023-02-10 13:54:11,780 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,780 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:11,780 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:11,780 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,781 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:11,781 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:11,783 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,783 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:11,783 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,784 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:11,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-10 13:54:11,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4251adb4[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:11,786 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-10 13:54:11,786 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:11,787 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,787 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:11,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:11,789 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:11,790 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:11,791 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:11,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:11,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-10 13:54:11,797 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-10 13:54:11,797 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:11,797 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:11,797 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-10 13:54:11,797 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:11,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:11,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:11,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:11,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:11,802 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:11,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,802 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,802 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:11,803 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,803 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,805 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,805 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,805 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:11,808 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-10 13:54:11,808 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:11,808 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:11,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,809 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:11,809 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:11,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,810 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,910 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:11,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:11,929 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:11,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:11,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-10 13:54:11,933 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-10 13:54:11,933 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,933 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-10 13:54:11,933 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-10 13:54:11,934 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,934 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,935 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,935 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,935 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:11,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:11,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:11,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:11,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:11,937 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:11,937 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:11,939 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:11,939 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. started... 2023-02-10 13:54:11,940 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,943 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:11,944 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,944 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:11,950 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 2023-02-10 13:54:11,970 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/322e9403a4d9470f8e68b3eb84dd3de4 2023-02-10 13:54:11,976 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 2023-02-10 13:54:11,982 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:11,983 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/322e9403a4d9470f8e68b3eb84dd3de4 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 2023-02-10 13:54:11,988 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:11,989 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 50ms, sequenceid=19, compaction requested=false 2023-02-10 13:54:11,989 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 763a2d94ec8c0a841c2f6b5578c7623c: 2023-02-10 13:54:11,989 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:11,989 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:11,991 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:11,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-10 13:54:11,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19972ccf[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:11,993 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-10 13:54:11,993 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,994 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,994 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,994 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,995 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:11,995 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:11,995 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,001 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,001 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,001 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,002 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:12,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-10 13:54:12,006 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-10 13:54:12,006 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:12,006 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:12,006 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-10 13:54:12,006 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:12,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,008 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,009 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,011 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,012 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,012 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,015 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,016 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,015 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,016 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,015 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,016 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-10 13:54:12,016 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:12,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,016 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:12,017 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:12,016 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,016 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,016 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,016 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,020 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,117 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:12,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:12,135 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:12,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:12,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-10 13:54:12,137 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-10 13:54:12,137 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,138 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-10 13:54:12,138 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-10 13:54:12,139 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,139 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,141 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,141 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,141 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,141 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,142 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,142 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,143 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,143 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,144 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,145 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,145 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:12,146 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. started... 2023-02-10 13:54:12,146 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,157 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/5b2d3bd09053407b9938c6bd31611487 2023-02-10 13:54:12,170 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/faad51426fc943a7b0f1223f669d5468 2023-02-10 13:54:12,176 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/5b2d3bd09053407b9938c6bd31611487 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 2023-02-10 13:54:12,181 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:12,182 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/faad51426fc943a7b0f1223f669d5468 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 2023-02-10 13:54:12,187 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:12,188 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 42ms, sequenceid=32, compaction requested=true 2023-02-10 13:54:12,188 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 763a2d94ec8c0a841c2f6b5578c7623c: 2023-02-10 13:54:12,188 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,188 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,190 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,190 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-10 13:54:12,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b71adcd[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:12,193 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-10 13:54:12,193 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,194 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,194 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,194 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,197 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,197 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:12,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-10 13:54:12,201 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-10 13:54:12,201 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:12,201 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:12,201 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-10 13:54:12,201 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:12,202 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,202 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,203 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,203 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,204 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,204 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,205 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,210 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,210 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,211 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,215 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,215 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,215 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,215 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,215 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,216 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,216 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,215 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,216 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-10 13:54:12,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,216 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:12,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,216 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,216 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,216 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,217 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,317 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:12,337 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:12,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:12,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-10 13:54:12,340 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-10 13:54:12,340 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,341 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-10 13:54:12,341 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-10 13:54:12,341 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,341 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,342 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,343 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,343 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:12,347 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,347 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. started... 2023-02-10 13:54:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,349 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:12,349 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,353 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,355 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,355 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,355 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,355 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,362 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/38ea83a673d140beae543ac87d705b04 2023-02-10 13:54:12,377 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/c482635a37474941b989067772013e9c 2023-02-10 13:54:12,383 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/38ea83a673d140beae543ac87d705b04 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 2023-02-10 13:54:12,388 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:12,389 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/c482635a37474941b989067772013e9c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c 2023-02-10 13:54:12,394 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:12,395 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 48ms, sequenceid=45, compaction requested=true 2023-02-10 13:54:12,395 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 763a2d94ec8c0a841c2f6b5578c7623c: 2023-02-10 13:54:12,395 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,395 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,397 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,397 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,397 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,397 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-10 13:54:12,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f0f8e41[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:12,399 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-10 13:54:12,399 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,401 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:12,401 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:12,403 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,405 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:12,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-10 13:54:12,408 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-10 13:54:12,408 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:12,408 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:12,408 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-10 13:54:12,408 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,410 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,411 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,411 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,412 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,414 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,414 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,414 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,418 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,418 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-10 13:54:12,418 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,418 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:12,419 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,419 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:12,519 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:12,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:12,538 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:12,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:12,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-10 13:54:12,540 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-10 13:54:12,540 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,540 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-10 13:54:12,540 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-10 13:54:12,541 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,541 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,542 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,542 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,542 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,542 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,543 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,543 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,544 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:12,545 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,545 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. started... 2023-02-10 13:54:12,545 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,545 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,562 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/d26eb406872b4d42bf7fd7723eef6760 2023-02-10 13:54:12,576 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/0821bcc488dc439e8078f8faf4d67805 2023-02-10 13:54:12,582 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/d26eb406872b4d42bf7fd7723eef6760 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 2023-02-10 13:54:12,588 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:12,589 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/0821bcc488dc439e8078f8faf4d67805 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 2023-02-10 13:54:12,595 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:12,596 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 51ms, sequenceid=58, compaction requested=true 2023-02-10 13:54:12,596 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 763a2d94ec8c0a841c2f6b5578c7623c: 2023-02-10 13:54:12,596 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,597 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,598 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,598 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,598 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,598 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-10 13:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407cbe19[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:12,601 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-10 13:54:12,601 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:12,602 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:12,602 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,610 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,611 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,611 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,611 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,611 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:12,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-10 13:54:12,614 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-10 13:54:12,614 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:12,614 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:12,614 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-10 13:54:12,614 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,616 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,618 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,619 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,620 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,621 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,621 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,627 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-10 13:54:12,627 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,627 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,627 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,628 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,629 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,629 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,629 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,629 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,728 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:12,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:12,748 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:12,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:12,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-10 13:54:12,750 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-10 13:54:12,750 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,750 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-10 13:54:12,750 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-10 13:54:12,751 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,751 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,752 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,752 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,752 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,752 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,752 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,752 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-10 13:54:12,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-10 13:54:12,754 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:12,755 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,755 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. started... 2023-02-10 13:54:12,755 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,767 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/f9590e14ca14442dbe040d21b2e9b8b9 2023-02-10 13:54:12,782 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/14ba6091a7754f808d6e3a542a85f8fc 2023-02-10 13:54:12,787 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/f9590e14ca14442dbe040d21b2e9b8b9 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 2023-02-10 13:54:12,792 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:12,793 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/14ba6091a7754f808d6e3a542a85f8fc as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc 2023-02-10 13:54:12,798 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:12,799 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 44ms, sequenceid=71, compaction requested=true 2023-02-10 13:54:12,799 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 763a2d94ec8c0a841c2f6b5578c7623c: 2023-02-10 13:54:12,799 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-10 13:54:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-10 13:54:12,802 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,802 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,802 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,802 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-10 13:54:12,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ad138d[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:12,804 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-10 13:54:12,804 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,805 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,805 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,805 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,806 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,806 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:12,806 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,807 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-10 13:54:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,808 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-10 13:54:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:12,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-10 13:54:12,811 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-10 13:54:12,812 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:12,812 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:12,812 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-10 13:54:12,812 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:12,814 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,814 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,814 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,814 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:12,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,814 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,814 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,815 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,815 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,815 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,816 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:12,817 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-10 13:54:12,818 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,818 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,821 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-10 13:54:12,821 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,821 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:12,821 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-10 13:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-10 13:54:12,822 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-10 13:54:12,922 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:12,937 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-10 13:54:12,937 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-10 13:54:12,938 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:12,938 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro Metrics about HBase RegionObservers 2023-02-10 13:54:13,186 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-02-10 13:54:13,187 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:13,189 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,190 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,190 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,190 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,193 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:13,193 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:13,194 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:13,439 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,440 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:13,441 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:13,441 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:13,443 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-02-10 13:54:13,444 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:13,444 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:13,444 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:13,444 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:13,445 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:13,448 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-02-10 13:54:13,449 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:13,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:13,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:13,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:13,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:13,452 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,452 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,452 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:13,453 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,453 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:13,453 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,453 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,453 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,454 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-02-10 13:54:13,454 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:13,455 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:13,459 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037283459 2023-02-10 13:54:13,461 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:13,461 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,464 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:13,465 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:13,465 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:13,465 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037313465 2023-02-10 13:54:13,465 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037373465 2023-02-10 13:54:13,465 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-10 13:54:13,465 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:13,467 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:13,467 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:13,467 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:13,715 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,715 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:13,716 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:13,716 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:13,717 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:13,717 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:13,717 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:13,717 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:13,717 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:13,718 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:13,719 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-10 13:54:13,720 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:13,720 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:13,720 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:13,721 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:13,721 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,721 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:13,722 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,722 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,722 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:13,723 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,723 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:13,723 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,723 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,723 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,723 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:13,724 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:13,724 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:13,725 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037283725 2023-02-10 13:54:13,725 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:13,726 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,729 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:13,729 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:13,730 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:13,730 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037313730 2023-02-10 13:54:13,730 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037373730 2023-02-10 13:54:13,730 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-10 13:54:13,730 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:13,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN}] 2023-02-10 13:54:13,733 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:13,733 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:13,733 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:13,980 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,980 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:13,980 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:13,980 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:13,981 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:13,981 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:13,981 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:13,981 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:13,981 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:13,982 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:13,983 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-10 13:54:13,983 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:13,984 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:13,984 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:13,984 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:13,985 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,985 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:13,986 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,986 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:13,986 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:13,986 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,987 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:13,987 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,987 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,987 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,987 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:13,987 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:13,988 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:13,988 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037283988 2023-02-10 13:54:13,990 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:13,990 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:13,993 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:13,993 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:13,993 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:13,993 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037313993 2023-02-10 13:54:13,993 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037373993 2023-02-10 13:54:13,993 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:13,994 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN}] 2023-02-10 13:54:13,994 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:13,996 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:13,997 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:13,997 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037253996"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037253996"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037253996"}]},"ts":"1676037253996"} 2023-02-10 13:54:13,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:13,998 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:13,998 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:13,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:13,998 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:14,244 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:14,244 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:14,245 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:14,246 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:14,247 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-10 13:54:14,247 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:14,247 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:14,247 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:14,248 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:14,248 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,249 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:14,249 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,249 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,249 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:14,250 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,250 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,250 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:14,250 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,250 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,251 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,251 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:14,251 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,251 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:14,251 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:14,252 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:14,252 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c to restore RIT 2023-02-10 13:54:14,252 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037284252 2023-02-10 13:54:14,253 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:14,253 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,256 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:14,256 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSING, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:14,256 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:14,256 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037314256 2023-02-10 13:54:14,256 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037374256 2023-02-10 13:54:14,256 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:14,257 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:14,257 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037254257"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037254257"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037254257"}]},"ts":"1676037254257"} 2023-02-10 13:54:14,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:14,259 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:14,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:14,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 763a2d94ec8c0a841c2f6b5578c7623c, disabling compactions & flushes 2023-02-10 13:54:14,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:14,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:14,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. after waiting 0 ms 2023-02-10 13:54:14,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:14,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 763a2d94ec8c0a841c2f6b5578c7623c 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:54:14,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/8eab5d9731644b67809a03a6e6ab9d2c 2023-02-10 13:54:14,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe 2023-02-10 13:54:14,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf1/8eab5d9731644b67809a03a6e6ab9d2c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c 2023-02-10 13:54:14,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:54:14,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/.tmp/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe 2023-02-10 13:54:14,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:54:14,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 48ms, sequenceid=82, compaction requested=true 2023-02-10 13:54:14,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:54:14,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:14,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. 2023-02-10 13:54:14,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 763a2d94ec8c0a841c2f6b5578c7623c: Waiting for close lock at 1676037254412Running coprocessor pre-close hooks at 1676037254412Disabling compacts and flushes for region at 1676037254412Disabling writes for close at 1676037254412Obtaining lock to block concurrent updates at 1676037254412Preparing flush snapshotting stores in 763a2d94ec8c0a841c2f6b5578c7623c at 1676037254412Finished memstore snapshotting testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1676037254412Flushing stores of testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. at 1676037254413 (+1 ms)Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf1: creating writer at 1676037254414 (+1 ms)Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf1: appending metadata at 1676037254416 (+2 ms)Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf1: closing flushed file at 1676037254416Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf2: creating writer at 1676037254430 (+14 ms)Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf2: appending metadata at 1676037254432 (+2 ms)Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf2: closing flushed file at 1676037254432Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf1: reopening flushed file at 1676037254447 (+15 ms)Flushing 763a2d94ec8c0a841c2f6b5578c7623c/cf2: reopening flushed file at 1676037254454 (+7 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 763a2d94ec8c0a841c2f6b5578c7623c in 48ms, sequenceid=82, compaction requested=true at 1676037254460 (+6 ms)Writing region close event to WAL at 1676037254478 (+18 ms)Running coprocessor post-close hooks at 1676037254483 (+5 ms)Closed at 1676037254483 2023-02-10 13:54:14,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:14,486 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED 2023-02-10 13:54:14,486 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037254486"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037254486"}]},"ts":"1676037254486"} 2023-02-10 13:54:14,487 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:14,487 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:14,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:14,487 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:14,507 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:14,507 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:14,508 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:14,509 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:14,509 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-10 13:54:14,509 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-02-10 13:54:14,510 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:14,510 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:14,510 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:14,510 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:14,511 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,511 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:14,511 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,511 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,512 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:14,513 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,513 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,513 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:14,513 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:14,513 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,513 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,513 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,513 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:14,513 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,513 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:14,513 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:14,514 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:14,514 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:14,515 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c to restore RIT 2023-02-10 13:54:14,515 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037284515 2023-02-10 13:54:14,515 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:14,515 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:14,518 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:14,518 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:14,518 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:14,518 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037314518 2023-02-10 13:54:14,518 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037374518 2023-02-10 13:54:14,518 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:14,519 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:14,519 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037254518"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037254518"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037254518"}]},"ts":"1676037254518"} 2023-02-10 13:54:14,520 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:14,523 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-02-10 13:54:14,523 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 in 261 msec 2023-02-10 13:54:14,524 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:14,524 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,524 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:14,524 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:14,769 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:14,769 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:14,770 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:14,771 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:14,771 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-10 13:54:14,772 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:14,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:14,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:14,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:14,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:14,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:14,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:14,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:14,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:14,775 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:14,775 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,776 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,776 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,776 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:14,776 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:14,776 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:14,776 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:14,777 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:14,777 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c to restore RIT 2023-02-10 13:54:14,777 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037284777 2023-02-10 13:54:14,778 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,778 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:14,780 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:14,781 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:14,781 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:14,781 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037314781 2023-02-10 13:54:14,781 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037374781 2023-02-10 13:54:14,781 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:14,781 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:14,783 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:14,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-02-10 13:54:14,786 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN in 789 msec 2023-02-10 13:54:14,789 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:14,789 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:14,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:14,789 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:15,031 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,031 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:15,032 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:15,032 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:15,032 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:15,032 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:15,032 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:15,032 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:15,032 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:15,033 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:15,034 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:15,034 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:15,035 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:15,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:15,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:15,036 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:15,036 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:15,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:15,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:15,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:15,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:15,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:15,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:15,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:15,039 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:15,039 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:15,039 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:15,039 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:15,039 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:15,040 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:15,040 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037285040 2023-02-10 13:54:15,042 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:15,042 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,044 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:15,044 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:15,044 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:15,044 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037315044 2023-02-10 13:54:15,045 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037375045 2023-02-10 13:54:15,045 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:15,045 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,045 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,045 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,049 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:15,059 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=763a2d94ec8c0a841c2f6b5578c7623c, threads=14 2023-02-10 13:54:15,060 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,060 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,060 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,060 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,061 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,061 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,061 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,061 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,061 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,062 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,062 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,062 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,062 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,062 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,196 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,199 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,202 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,205 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,215 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,219 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,222 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,225 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,240 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,241 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,244 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,246 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,567 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,628 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,628 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region 763a2d94ec8c0a841c2f6b5578c7623c Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:54:15,666 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:15,666 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:15,667 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:15,795 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,795 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:15,795 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:15,795 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:15,796 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:15,796 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:15,796 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:15,796 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:15,796 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:15,797 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:15,798 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-10 13:54:15,798 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:15,798 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:15,798 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:15,799 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:15,799 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:15,799 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:15,800 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:15,800 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:15,801 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:15,801 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:15,802 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:15,802 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,802 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:15,803 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:15,803 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:15,803 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,803 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,803 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,803 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:15,803 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:15,803 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-10 13:54:15,804 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:15,804 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:15,804 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037285804 2023-02-10 13:54:15,805 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:15,806 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:15,809 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:15,809 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:15,809 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:15,809 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037315809 2023-02-10 13:54:15,809 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037375809 2023-02-10 13:54:15,809 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:15,820 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=763a2d94ec8c0a841c2f6b5578c7623c, threads=14 2023-02-10 13:54:15,822 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,822 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,822 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,823 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,823 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,823 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,823 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,824 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,824 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,824 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,824 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,824 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,824 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,828 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,978 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,979 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,979 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,984 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,984 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:15,992 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,001 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,008 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,008 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,016 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,017 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,017 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,286 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:54:16,349 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,356 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c for region: 763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:16,356 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region 763a2d94ec8c0a841c2f6b5578c7623c Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:54:16,395 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:16,400 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:54:16,402 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:54:16,402 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:16,402 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:16,402 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:16,560 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,560 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:16,560 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:16,560 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:16,561 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:16,561 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:16,561 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:16,561 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:16,561 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:16,562 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:16,563 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-10 13:54:16,563 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:16,563 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:16,563 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:16,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:16,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:16,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:16,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:16,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:16,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:16,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:16,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:16,566 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,566 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:16,566 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:16,566 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:16,566 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,566 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,566 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,567 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:16,567 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:16,567 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:16,567 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:16,568 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:16,568 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037286568 2023-02-10 13:54:16,568 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:16,570 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,572 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:16,573 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:16,573 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:16,573 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037316573 2023-02-10 13:54:16,573 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037376573 2023-02-10 13:54:16,573 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-10 13:54:16,577 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:16,579 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:16,579 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,579 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:16,579 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:16,823 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:16,824 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:16,824 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:16,825 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:16,826 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-10 13:54:16,826 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:16,826 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:16,826 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:16,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:16,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:16,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:16,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:16,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:16,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:16,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:16,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:16,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN 2023-02-10 13:54:16,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:16,829 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:16,829 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,829 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,829 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,829 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=763a2d94ec8c0a841c2f6b5578c7623c, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:16,829 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 763a2d94ec8c0a841c2f6b5578c7623c, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:16,830 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:16,830 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:16,831 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:16,831 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037286831 2023-02-10 13:54:16,833 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:16,833 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:16,835 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:16,835 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=763a2d94ec8c0a841c2f6b5578c7623c, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:16,835 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:16,835 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037316835 2023-02-10 13:54:16,835 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037376835 2023-02-10 13:54:16,835 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:16,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037256838"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1676037256838"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1676037256838"}]},"ts":"1676037256838"} 2023-02-10 13:54:16,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037256838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037256838"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037256838"}]},"ts":"1676037256838"} 2023-02-10 13:54:16,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037256838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037256838"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037256838"}]},"ts":"1676037256838"} 2023-02-10 13:54:16,845 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, ASSIGN}, {pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, ASSIGN}] 2023-02-10 13:54:16,846 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, ASSIGN 2023-02-10 13:54:16,847 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:16,848 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, ASSIGN 2023-02-10 13:54:16,849 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:16,997 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:54:16,999 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=849462bcb81926d6eef14ec7231a1129, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:16,999 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037256999"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037256999"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037256999"}]},"ts":"1676037256999"} 2023-02-10 13:54:17,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure 849462bcb81926d6eef14ec7231a1129, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:17,005 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=eda211d0e52e267866f099e27926757d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:17,005 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257005"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037257005"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037257005"}]},"ts":"1676037257005"} 2023-02-10 13:54:17,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=32, state=RUNNABLE; OpenRegionProcedure eda211d0e52e267866f099e27926757d, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:17,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 849462bcb81926d6eef14ec7231a1129, NAME => 'testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.', STARTKEY => '', ENDKEY => '41'} 2023-02-10 13:54:17,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:17,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:17,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,160 INFO [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,160 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1 2023-02-10 13:54:17,160 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1 2023-02-10 13:54:17,161 INFO [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 849462bcb81926d6eef14ec7231a1129 columnFamilyName cf1 2023-02-10 13:54:17,177 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04-bottom 2023-02-10 13:54:17,181 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487-bottom 2023-02-10 13:54:17,187 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c-bottom 2023-02-10 13:54:17,192 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b-bottom 2023-02-10 13:54:17,197 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760-bottom 2023-02-10 13:54:17,202 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4-bottom 2023-02-10 13:54:17,207 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9-bottom 2023-02-10 13:54:17,207 INFO [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(310): Store=849462bcb81926d6eef14ec7231a1129/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:17,208 INFO [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,209 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2 2023-02-10 13:54:17,209 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2 2023-02-10 13:54:17,209 INFO [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 849462bcb81926d6eef14ec7231a1129 columnFamilyName cf2 2023-02-10 13:54:17,226 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805-bottom 2023-02-10 13:54:17,233 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc-bottom 2023-02-10 13:54:17,242 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4-bottom 2023-02-10 13:54:17,247 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c-bottom 2023-02-10 13:54:17,252 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c-bottom 2023-02-10 13:54:17,258 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe-bottom 2023-02-10 13:54:17,263 DEBUG [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468-bottom 2023-02-10 13:54:17,263 INFO [StoreOpener-849462bcb81926d6eef14ec7231a1129-1] regionserver.HStore(310): Store=849462bcb81926d6eef14ec7231a1129/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:17,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:17,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 849462bcb81926d6eef14ec7231a1129; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68862958, jitterRate=0.02613803744316101}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:17,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 849462bcb81926d6eef14ec7231a1129: Running coprocessor pre-open hook at 1676037257158Writing region info on filesystem at 1676037257158Initializing all the Stores at 1676037257159 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037257159Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037257159Cleaning up temporary data from old regions at 1676037257265 (+106 ms)Cleaning up detritus from prior splits at 1676037257266 (+1 ms)Running coprocessor post-open hooks at 1676037257268 (+2 ms)Region opened successfully at 1676037257268 2023-02-10 13:54:17,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129., pid=33, masterSystemTime=1676037257153 2023-02-10 13:54:17,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:17,270 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:17,270 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:17,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:17,270 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,270 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,270 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1912): 849462bcb81926d6eef14ec7231a1129/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:17,271 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1912): 849462bcb81926d6eef14ec7231a1129/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:17,271 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 849462bcb81926d6eef14ec7231a1129/cf2 in testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,271 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 849462bcb81926d6eef14ec7231a1129/cf1 in testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,271 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/.tmp, totalSize=34.9 K 2023-02-10 13:54:17,271 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/.tmp, totalSize=34.9 K 2023-02-10 13:54:17,272 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037251715 2023-02-10 13:54:17,272 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037251715 2023-02-10 13:54:17,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eda211d0e52e267866f099e27926757d, NAME => 'testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.', STARTKEY => '41', ENDKEY => ''} 2023-02-10 13:54:17,273 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037251912 2023-02-10 13:54:17,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:17,274 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037251912 2023-02-10 13:54:17,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,274 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037252119 2023-02-10 13:54:17,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:17,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,274 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=849462bcb81926d6eef14ec7231a1129, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:17,275 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257274"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037257274"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037257274"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037257274"}]},"ts":"1676037257274"} 2023-02-10 13:54:17,277 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037252319 2023-02-10 13:54:17,277 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037252119 2023-02-10 13:54:17,277 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037252521 2023-02-10 13:54:17,278 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037252319 2023-02-10 13:54:17,278 INFO [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,278 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037252731 2023-02-10 13:54:17,279 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037252521 2023-02-10 13:54:17,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-02-10 13:54:17,279 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037252924 2023-02-10 13:54:17,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure 849462bcb81926d6eef14ec7231a1129, server=jenkins-hbase4.apache.org,46307,1676037243975 in 275 msec 2023-02-10 13:54:17,279 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037252731 2023-02-10 13:54:17,280 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1 2023-02-10 13:54:17,280 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1 2023-02-10 13:54:17,280 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037252924 2023-02-10 13:54:17,281 INFO [StoreOpener-eda211d0e52e267866f099e27926757d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eda211d0e52e267866f099e27926757d columnFamilyName cf1 2023-02-10 13:54:17,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, ASSIGN in 435 msec 2023-02-10 13:54:17,301 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] throttle.PressureAwareThroughputController(145): 849462bcb81926d6eef14ec7231a1129#cf2#compaction#32 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:17,303 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04-top 2023-02-10 13:54:17,304 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] throttle.PressureAwareThroughputController(145): 849462bcb81926d6eef14ec7231a1129#cf1#compaction#33 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:17,323 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487-top 2023-02-10 13:54:17,328 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c-top 2023-02-10 13:54:17,332 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b-top 2023-02-10 13:54:17,336 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760-top 2023-02-10 13:54:17,340 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4-top 2023-02-10 13:54:17,344 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9-top 2023-02-10 13:54:17,350 INFO [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(310): Store=eda211d0e52e267866f099e27926757d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:17,350 INFO [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,351 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2 2023-02-10 13:54:17,351 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2 2023-02-10 13:54:17,352 INFO [StoreOpener-eda211d0e52e267866f099e27926757d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eda211d0e52e267866f099e27926757d columnFamilyName cf2 2023-02-10 13:54:17,363 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/.tmp/cf2/4d033a3aee714f1394b485ae1a9b1229 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/4d033a3aee714f1394b485ae1a9b1229 2023-02-10 13:54:17,365 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/.tmp/cf1/bc0213cad6764c17a431d26bd5e83945 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/bc0213cad6764c17a431d26bd5e83945 2023-02-10 13:54:17,370 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 849462bcb81926d6eef14ec7231a1129/cf2 of 849462bcb81926d6eef14ec7231a1129 into 4d033a3aee714f1394b485ae1a9b1229(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:17,370 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 849462bcb81926d6eef14ec7231a1129: 2023-02-10 13:54:17,370 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129., storeName=849462bcb81926d6eef14ec7231a1129/cf2, priority=9, startTime=1676037257270; duration=0sec 2023-02-10 13:54:17,371 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:17,373 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 849462bcb81926d6eef14ec7231a1129/cf1 of 849462bcb81926d6eef14ec7231a1129 into bc0213cad6764c17a431d26bd5e83945(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:17,373 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 849462bcb81926d6eef14ec7231a1129: 2023-02-10 13:54:17,373 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129., storeName=849462bcb81926d6eef14ec7231a1129/cf1, priority=9, startTime=1676037257269; duration=0sec 2023-02-10 13:54:17,373 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:17,380 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805-top 2023-02-10 13:54:17,384 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc-top 2023-02-10 13:54:17,389 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4-top 2023-02-10 13:54:17,393 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c-top 2023-02-10 13:54:17,401 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c-top 2023-02-10 13:54:17,406 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe-top 2023-02-10 13:54:17,411 DEBUG [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468-top 2023-02-10 13:54:17,411 INFO [StoreOpener-eda211d0e52e267866f099e27926757d-1] regionserver.HStore(310): Store=eda211d0e52e267866f099e27926757d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:17,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:17,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened eda211d0e52e267866f099e27926757d; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68462201, jitterRate=0.020166292786598206}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:17,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for eda211d0e52e267866f099e27926757d: Running coprocessor pre-open hook at 1676037257274Writing region info on filesystem at 1676037257274Initializing all the Stores at 1676037257277 (+3 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037257277Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037257278 (+1 ms)Cleaning up temporary data from old regions at 1676037257413 (+135 ms)Cleaning up detritus from prior splits at 1676037257414 (+1 ms)Running coprocessor post-open hooks at 1676037257416 (+2 ms)Region opened successfully at 1676037257416 2023-02-10 13:54:17,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d., pid=34, masterSystemTime=1676037257153 2023-02-10 13:54:17,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:17,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:17,418 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:17,418 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:17,418 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,418 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1912): eda211d0e52e267866f099e27926757d/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:17,418 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,418 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1912): eda211d0e52e267866f099e27926757d/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:17,418 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of eda211d0e52e267866f099e27926757d/cf1 in testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,418 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegion(2258): Starting compaction of eda211d0e52e267866f099e27926757d/cf2 in testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,419 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c-top] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/.tmp, totalSize=34.9 K 2023-02-10 13:54:17,419 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe-top] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/.tmp, totalSize=34.9 K 2023-02-10 13:54:17,419 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037251715 2023-02-10 13:54:17,419 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037251715 2023-02-10 13:54:17,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,420 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037251912 2023-02-10 13:54:17,420 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037251912 2023-02-10 13:54:17,420 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=eda211d0e52e267866f099e27926757d, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:17,420 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037252119 2023-02-10 13:54:17,421 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257420"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037257420"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037257420"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037257420"}]},"ts":"1676037257420"} 2023-02-10 13:54:17,421 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037252119 2023-02-10 13:54:17,421 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037252319 2023-02-10 13:54:17,421 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037252319 2023-02-10 13:54:17,422 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1676037252521 2023-02-10 13:54:17,423 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1676037252521 2023-02-10 13:54:17,423 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1676037252731 2023-02-10 13:54:17,424 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1676037252731 2023-02-10 13:54:17,424 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037252924 2023-02-10 13:54:17,425 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037252924 2023-02-10 13:54:17,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=32 2023-02-10 13:54:17,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=32, state=SUCCESS; OpenRegionProcedure eda211d0e52e267866f099e27926757d, server=jenkins-hbase4.apache.org,46307,1676037243975 in 417 msec 2023-02-10 13:54:17,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-02-10 13:54:17,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, ASSIGN in 584 msec 2023-02-10 13:54:17,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d in 4.2430 sec 2023-02-10 13:54:17,440 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] throttle.PressureAwareThroughputController(145): eda211d0e52e267866f099e27926757d#cf1#compaction#34 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:17,443 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] throttle.PressureAwareThroughputController(145): eda211d0e52e267866f099e27926757d#cf2#compaction#35 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:17,467 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/.tmp/cf1/f7e2873bda7b4403b47ab096cda3075c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f7e2873bda7b4403b47ab096cda3075c 2023-02-10 13:54:17,469 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/.tmp/cf2/7180d904d8234d9c87cdec0c8a94a603 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/7180d904d8234d9c87cdec0c8a94a603 2023-02-10 13:54:17,473 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eda211d0e52e267866f099e27926757d/cf1 of eda211d0e52e267866f099e27926757d into f7e2873bda7b4403b47ab096cda3075c(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:17,474 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for eda211d0e52e267866f099e27926757d: 2023-02-10 13:54:17,474 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d., storeName=eda211d0e52e267866f099e27926757d/cf1, priority=9, startTime=1676037257417; duration=0sec 2023-02-10 13:54:17,474 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:17,475 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in eda211d0e52e267866f099e27926757d/cf2 of eda211d0e52e267866f099e27926757d into 7180d904d8234d9c87cdec0c8a94a603(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:17,475 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for eda211d0e52e267866f099e27926757d: 2023-02-10 13:54:17,475 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d., storeName=eda211d0e52e267866f099e27926757d/cf2, priority=9, startTime=1676037257418; duration=0sec 2023-02-10 13:54:17,475 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:17,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46307] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Scan size: 143 connection: 172.31.14.131:41738 deadline: 1676037317586, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c. is not online on jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:17,705 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:54:17,723 INFO [Listener at localhost/43477] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-10 13:54:17,725 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:54:17,750 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:17,752 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testSplitWithoutPONR 2023-02-10 13:54:17,752 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testSplitWithoutPONR 2023-02-10 13:54:17,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:17,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-02-10 13:54:17,757 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037257756"}]},"ts":"1676037257756"} 2023-02-10 13:54:17,758 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLING in hbase:meta 2023-02-10 13:54:17,759 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitWithoutPONR to state=DISABLING 2023-02-10 13:54:17,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, UNASSIGN}, {pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, UNASSIGN}] 2023-02-10 13:54:17,761 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, UNASSIGN 2023-02-10 13:54:17,762 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=eda211d0e52e267866f099e27926757d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:17,762 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257762"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037257762"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037257762"}]},"ts":"1676037257762"} 2023-02-10 13:54:17,764 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=37, state=RUNNABLE; CloseRegionProcedure eda211d0e52e267866f099e27926757d, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:17,766 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, UNASSIGN 2023-02-10 13:54:17,767 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=849462bcb81926d6eef14ec7231a1129, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:17,767 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257767"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037257767"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037257767"}]},"ts":"1676037257767"} 2023-02-10 13:54:17,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=36, state=RUNNABLE; CloseRegionProcedure 849462bcb81926d6eef14ec7231a1129, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:17,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-02-10 13:54:17,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing eda211d0e52e267866f099e27926757d, disabling compactions & flushes 2023-02-10 13:54:17,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. after waiting 0 ms 2023-02-10 13:54:17,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,929 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c-top] to archive 2023-02-10 13:54:17,930 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:17,932 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,934 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,935 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,936 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,938 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,939 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,940 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,942 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe-top] to archive 2023-02-10 13:54:17,943 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:17,945 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,946 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,947 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,948 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,950 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,951 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,951 DEBUG [StoreCloser-testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:54:17,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:17,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d. 2023-02-10 13:54:17,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for eda211d0e52e267866f099e27926757d: Waiting for close lock at 1676037257916Running coprocessor pre-close hooks at 1676037257916Disabling compacts and flushes for region at 1676037257916Disabling writes for close at 1676037257917 (+1 ms)Writing region close event to WAL at 1676037257953 (+36 ms)Running coprocessor post-close hooks at 1676037257956 (+3 ms)Closed at 1676037257956 2023-02-10 13:54:17,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed eda211d0e52e267866f099e27926757d 2023-02-10 13:54:17,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 849462bcb81926d6eef14ec7231a1129, disabling compactions & flushes 2023-02-10 13:54:17,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. after waiting 0 ms 2023-02-10 13:54:17,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,959 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=eda211d0e52e267866f099e27926757d, regionState=CLOSED 2023-02-10 13:54:17,959 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037257959"}]},"ts":"1676037257959"} 2023-02-10 13:54:17,960 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c-bottom] to archive 2023-02-10 13:54:17,960 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:17,962 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=37 2023-02-10 13:54:17,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=37, state=SUCCESS; CloseRegionProcedure eda211d0e52e267866f099e27926757d, server=jenkins-hbase4.apache.org,46307,1676037243975 in 197 msec 2023-02-10 13:54:17,965 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=eda211d0e52e267866f099e27926757d, UNASSIGN in 203 msec 2023-02-10 13:54:17,965 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,967 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,969 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,971 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,972 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,973 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,975 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe-bottom] to archive 2023-02-10 13:54:17,976 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:17,977 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,979 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,980 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,981 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,982 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,983 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,984 DEBUG [StoreCloser-testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:17,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:54:17,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:17,989 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129. 2023-02-10 13:54:17,989 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 849462bcb81926d6eef14ec7231a1129: Waiting for close lock at 1676037257958Running coprocessor pre-close hooks at 1676037257958Disabling compacts and flushes for region at 1676037257958Disabling writes for close at 1676037257958Writing region close event to WAL at 1676037257985 (+27 ms)Running coprocessor post-close hooks at 1676037257989 (+4 ms)Closed at 1676037257989 2023-02-10 13:54:17,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:17,991 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=849462bcb81926d6eef14ec7231a1129, regionState=CLOSED 2023-02-10 13:54:17,991 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037257991"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037257991"}]},"ts":"1676037257991"} 2023-02-10 13:54:17,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=36 2023-02-10 13:54:17,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; CloseRegionProcedure 849462bcb81926d6eef14ec7231a1129, server=jenkins-hbase4.apache.org,46307,1676037243975 in 224 msec 2023-02-10 13:54:17,997 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-02-10 13:54:17,997 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=849462bcb81926d6eef14ec7231a1129, UNASSIGN in 235 msec 2023-02-10 13:54:17,998 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037257998"}]},"ts":"1676037257998"} 2023-02-10 13:54:17,999 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLED in hbase:meta 2023-02-10 13:54:18,001 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitWithoutPONR to state=DISABLED 2023-02-10 13:54:18,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR in 249 msec 2023-02-10 13:54:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-02-10 13:54:18,059 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitWithoutPONR, procId: 35 completed 2023-02-10 13:54:18,060 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testSplitWithoutPONR 2023-02-10 13:54:18,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:18,063 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:18,064 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=40, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:18,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-02-10 13:54:18,072 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,072 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d 2023-02-10 13:54:18,072 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:18,074 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/recovered.edits] 2023-02-10 13:54:18,075 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/recovered.edits] 2023-02-10 13:54:18,075 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/recovered.edits] 2023-02-10 13:54:18,078 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f7e2873bda7b4403b47ab096cda3075c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf1/f7e2873bda7b4403b47ab096cda3075c 2023-02-10 13:54:18,079 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/38ea83a673d140beae543ac87d705b04 2023-02-10 13:54:18,079 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/.regioninfo to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/.regioninfo 2023-02-10 13:54:18,081 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/5b2d3bd09053407b9938c6bd31611487 2023-02-10 13:54:18,082 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/7180d904d8234d9c87cdec0c8a94a603 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/cf2/7180d904d8234d9c87cdec0c8a94a603 2023-02-10 13:54:18,082 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/8eab5d9731644b67809a03a6e6ab9d2c 2023-02-10 13:54:18,083 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,084 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/957aee19684d45b6b016046978054d2b 2023-02-10 13:54:18,085 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,086 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/.regioninfo to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/.regioninfo 2023-02-10 13:54:18,087 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/d26eb406872b4d42bf7fd7723eef6760 2023-02-10 13:54:18,087 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,089 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4 2023-02-10 13:54:18,089 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,091 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf1/f9590e14ca14442dbe040d21b2e9b8b9 2023-02-10 13:54:18,091 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/38ea83a673d140beae543ac87d705b04.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,092 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,093 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/5b2d3bd09053407b9938c6bd31611487.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,093 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,095 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,095 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/0821bcc488dc439e8078f8faf4d67805 2023-02-10 13:54:18,095 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/8eab5d9731644b67809a03a6e6ab9d2c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,108 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/957aee19684d45b6b016046978054d2b.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,109 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/14ba6091a7754f808d6e3a542a85f8fc 2023-02-10 13:54:18,112 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/d26eb406872b4d42bf7fd7723eef6760.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,112 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,112 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/322e9403a4d9470f8e68b3eb84dd3de4 2023-02-10 13:54:18,118 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/3b795b2576a2412cb61d6fd4eb168b5c 2023-02-10 13:54:18,119 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,119 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/f5e10011bb1c4bb5b0abdb7733d4fee4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,121 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/c482635a37474941b989067772013e9c 2023-02-10 13:54:18,121 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,122 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf1/f9590e14ca14442dbe040d21b2e9b8b9.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,123 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe 2023-02-10 13:54:18,125 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,125 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/cf2/faad51426fc943a7b0f1223f669d5468 2023-02-10 13:54:18,127 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,128 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/0821bcc488dc439e8078f8faf4d67805.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,129 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,130 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/14ba6091a7754f808d6e3a542a85f8fc.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,130 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/recovered.edits/85.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c/recovered.edits/85.seqid 2023-02-10 13:54:18,131 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,132 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/849462bcb81926d6eef14ec7231a1129/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,133 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/322e9403a4d9470f8e68b3eb84dd3de4.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,135 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/3b795b2576a2412cb61d6fd4eb168b5c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,136 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/bc0213cad6764c17a431d26bd5e83945 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf1/bc0213cad6764c17a431d26bd5e83945 2023-02-10 13:54:18,137 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/c482635a37474941b989067772013e9c.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,139 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/f5f4bcc6bcfd4d16b5b6026ae9960bfe.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,140 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/4d033a3aee714f1394b485ae1a9b1229 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/cf2/4d033a3aee714f1394b485ae1a9b1229 2023-02-10 13:54:18,141 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/eda211d0e52e267866f099e27926757d/cf2/faad51426fc943a7b0f1223f669d5468.763a2d94ec8c0a841c2f6b5578c7623c 2023-02-10 13:54:18,147 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129/recovered.edits/90.seqid 2023-02-10 13:54:18,147 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d/recovered.edits/90.seqid 2023-02-10 13:54:18,148 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/eda211d0e52e267866f099e27926757d 2023-02-10 13:54:18,149 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitWithoutPONR/849462bcb81926d6eef14ec7231a1129 2023-02-10 13:54:18,149 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-02-10 13:54:18,152 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=40, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:18,155 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitWithoutPONR from hbase:meta 2023-02-10 13:54:18,160 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitWithoutPONR' descriptor. 2023-02-10 13:54:18,162 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=40, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:18,162 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitWithoutPONR' from region states. 2023-02-10 13:54:18,162 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037258162"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:18,162 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037258162"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:18,162 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037258162"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:18,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-02-10 13:54:18,166 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:54:18,166 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 763a2d94ec8c0a841c2f6b5578c7623c, NAME => 'testSplitWithoutPONR,,1676037251205.763a2d94ec8c0a841c2f6b5578c7623c.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 849462bcb81926d6eef14ec7231a1129, NAME => 'testSplitWithoutPONR,,1676037253187.849462bcb81926d6eef14ec7231a1129.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => eda211d0e52e267866f099e27926757d, NAME => 'testSplitWithoutPONR,41,1676037253187.eda211d0e52e267866f099e27926757d.', STARTKEY => '41', ENDKEY => ''}] 2023-02-10 13:54:18,166 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitWithoutPONR' as deleted. 2023-02-10 13:54:18,166 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037258166"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:18,168 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitWithoutPONR state from META 2023-02-10 13:54:18,170 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=40, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:18,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR in 110 msec 2023-02-10 13:54:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-02-10 13:54:18,367 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitWithoutPONR, procId: 40 completed 2023-02-10 13:54:18,386 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=403 (was 390) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:49702 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:48398 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:48448 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:37998 [Waiting for operation #25] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:48414 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:47812 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28c66558-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:48436 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:1;jenkins-hbase4:46307-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:49732 [Waiting for operation #25] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:47848 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:37834 [Waiting for operation #38] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:42100 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:49768 [Waiting for operation #20] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:37732 [Waiting for operation #34] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:42176 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:48426 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x28c66558-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1447706916_15 at /127.0.0.1:48488 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=871 (was 833) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=270 (was 250) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3523 (was 3645) 2023-02-10 13:54:18,402 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=403, OpenFileDescriptor=871, MaxFileDescriptor=60000, SystemLoadAverage=270, ProcessCount=168, AvailableMemoryMB=3522 2023-02-10 13:54:18,402 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:18,405 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:54:18,406 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:54:18,407 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:18,409 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:18,410 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:18,412 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,412 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 empty. 2023-02-10 13:54:18,413 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,413 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-02-10 13:54:18,424 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:18,426 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9223b687f29223ce102064f5414860f7, NAME => 'testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:18,434 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(865): Instantiated testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:18,434 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1603): Closing 9223b687f29223ce102064f5414860f7, disabling compactions & flushes 2023-02-10 13:54:18,434 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1625): Closing region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,434 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,434 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1713): Acquired close lock on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. after waiting 0 ms 2023-02-10 13:54:18,434 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1723): Updates disabled for region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,434 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1837): Closed testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,434 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1557): Region close journal for 9223b687f29223ce102064f5414860f7: Waiting for close lock at 1676037258434Disabling compacts and flushes for region at 1676037258434Disabling writes for close at 1676037258434Writing region close event to WAL at 1676037258434Closed at 1676037258434 2023-02-10 13:54:18,437 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:18,438 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037258437"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037258437"}]},"ts":"1676037258437"} 2023-02-10 13:54:18,439 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:18,440 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:18,440 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037258440"}]},"ts":"1676037258440"} 2023-02-10 13:54:18,442 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLING in hbase:meta 2023-02-10 13:54:18,444 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:18,445 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:18,445 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:18,445 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:18,446 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:18,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN}] 2023-02-10 13:54:18,447 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:18,448 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:18,598 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:18,600 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,600 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037258599"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037258599"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037258599"}]},"ts":"1676037258599"} 2023-02-10 13:54:18,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:18,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9223b687f29223ce102064f5414860f7, NAME => 'testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:18,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:18,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:18,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,760 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,761 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1 2023-02-10 13:54:18,761 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1 2023-02-10 13:54:18,762 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9223b687f29223ce102064f5414860f7 columnFamilyName cf1 2023-02-10 13:54:18,762 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(310): Store=9223b687f29223ce102064f5414860f7/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:18,762 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,763 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2 2023-02-10 13:54:18,763 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2 2023-02-10 13:54:18,764 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9223b687f29223ce102064f5414860f7 columnFamilyName cf2 2023-02-10 13:54:18,764 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(310): Store=9223b687f29223ce102064f5414860f7/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:18,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:18,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:18,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:18,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9223b687f29223ce102064f5414860f7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67136075, jitterRate=4.054754972457886E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:18,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9223b687f29223ce102064f5414860f7: Running coprocessor pre-open hook at 1676037258759Writing region info on filesystem at 1676037258759Initializing all the Stores at 1676037258759Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037258759Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037258760 (+1 ms)Cleaning up temporary data from old regions at 1676037258766 (+6 ms)Cleaning up detritus from prior splits at 1676037258766Running coprocessor post-open hooks at 1676037258771 (+5 ms)Region opened successfully at 1676037258771 2023-02-10 13:54:18,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7., pid=43, masterSystemTime=1676037258754 2023-02-10 13:54:18,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,773 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,774 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037258773"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037258773"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037258773"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037258773"}]},"ts":"1676037258773"} 2023-02-10 13:54:18,776 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-10 13:54:18,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 in 173 msec 2023-02-10 13:54:18,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=41 2023-02-10 13:54:18,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=41, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN in 331 msec 2023-02-10 13:54:18,779 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:18,779 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037258779"}]},"ts":"1676037258779"} 2023-02-10 13:54:18,781 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLED in hbase:meta 2023-02-10 13:54:18,787 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:18,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution in 381 msec 2023-02-10 13:54:18,916 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-10 13:54:18,918 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-10 13:54:18,918 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:18,918 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-10 13:54:18,918 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:54:18,919 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,919 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:18,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:18,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:18,920 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:18,920 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,920 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:18,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:18,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:18,922 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,922 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. started... 2023-02-10 13:54:18,923 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:54:18,925 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,926 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,926 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,926 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:18,926 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,926 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:18,937 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/39d92a278982404a8fabb4e6fad7907f 2023-02-10 13:54:18,950 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/1a3e690bb1014153bc2ccef997bd7e98 2023-02-10 13:54:18,955 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/39d92a278982404a8fabb4e6fad7907f as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f 2023-02-10 13:54:18,959 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:18,960 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/1a3e690bb1014153bc2ccef997bd7e98 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 2023-02-10 13:54:18,964 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:18,965 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 42ms, sequenceid=6, compaction requested=false 2023-02-10 13:54:18,965 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-02-10 13:54:18,966 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:18,966 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:18,966 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:18,967 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:18,968 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,968 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-10 13:54:18,971 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d4c590[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:18,971 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:18,972 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,972 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:18,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:18,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:18,973 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:18,974 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:18,974 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:18,974 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:18,975 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:18,975 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,975 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:18,975 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:18,975 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:18,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:18,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:18,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:54:18,979 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-10 13:54:18,979 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:18,979 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:18,979 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-10 13:54:18,979 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:18,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:18,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:18,981 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:18,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:18,982 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:18,982 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:18,984 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:18,984 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,984 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:18,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:18,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,987 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:18,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:18,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:18,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,987 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-10 13:54:18,988 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:18,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:18,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:18,988 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:18,989 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:18,989 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:18,989 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,089 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:19,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:19,105 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:19,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:19,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-10 13:54:19,106 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,106 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,106 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-10 13:54:19,107 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:54:19,107 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,107 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,109 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,109 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,109 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,109 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,110 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,110 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,111 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:19,112 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,112 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. started... 2023-02-10 13:54:19,112 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:19,113 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,114 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,115 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,124 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/32772e7cd7a74abea498a01369f95e5b 2023-02-10 13:54:19,136 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/fa7e61d1f1f6472caa52d191eb388dd2 2023-02-10 13:54:19,142 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/32772e7cd7a74abea498a01369f95e5b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b 2023-02-10 13:54:19,149 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:19,150 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/fa7e61d1f1f6472caa52d191eb388dd2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 2023-02-10 13:54:19,155 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:19,156 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 44ms, sequenceid=19, compaction requested=false 2023-02-10 13:54:19,157 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:19,157 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,157 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:19,157 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:19,157 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,157 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,157 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,159 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-10 13:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75da3e8d[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:19,161 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:54:19,161 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,164 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,164 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,168 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:19,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:54:19,171 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-10 13:54:19,171 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:19,171 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:19,171 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-10 13:54:19,171 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,172 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,173 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,173 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,174 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,175 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,176 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,176 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,179 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,179 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,180 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,180 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:19,180 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,180 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,179 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,179 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,180 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:19,180 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:19,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,180 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,180 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,181 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,280 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:19,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:19,297 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:19,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:19,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-10 13:54:19,299 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-10 13:54:19,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:54:19,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,301 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,301 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,301 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,301 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,301 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,301 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:19,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:19,304 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:19,304 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. started... 2023-02-10 13:54:19,304 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,314 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/fbb4f2f534f1495bad519f870de9d2c6 2023-02-10 13:54:19,326 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/d121748ce713426d969589a390a13ce3 2023-02-10 13:54:19,330 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/fbb4f2f534f1495bad519f870de9d2c6 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 2023-02-10 13:54:19,334 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:19,335 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/d121748ce713426d969589a390a13ce3 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 2023-02-10 13:54:19,338 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:19,339 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 35ms, sequenceid=32, compaction requested=true 2023-02-10 13:54:19,339 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:19,339 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:19,340 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:19,340 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,340 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,340 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,341 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,341 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-10 13:54:19,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f3ca59[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:19,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:54:19,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,345 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,346 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:19,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:19,347 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,348 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:19,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:54:19,352 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-10 13:54:19,352 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:19,352 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:19,352 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-10 13:54:19,352 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:19,353 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,354 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,353 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,353 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,355 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,355 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,358 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,358 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,358 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,365 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,365 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,365 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:19,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,366 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,365 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,366 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,466 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:19,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:19,482 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-10 13:54:19,483 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,483 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,483 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-10 13:54:19,484 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:54:19,484 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,484 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,485 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,485 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,485 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,485 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,485 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,485 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:19,487 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,487 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. started... 2023-02-10 13:54:19,488 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:19,488 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,491 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,491 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,491 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,491 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,491 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,497 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/1488f2ad74af485caccfd05315c5be90 2023-02-10 13:54:19,509 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/518bfa8b4be84d3984404c2670636ebb 2023-02-10 13:54:19,513 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/1488f2ad74af485caccfd05315c5be90 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 2023-02-10 13:54:19,518 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:19,519 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/518bfa8b4be84d3984404c2670636ebb as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb 2023-02-10 13:54:19,523 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:19,524 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 37ms, sequenceid=45, compaction requested=true 2023-02-10 13:54:19,524 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:19,524 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,524 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:19,524 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:19,524 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,524 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,524 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,526 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-10 13:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ca9ab1[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:19,528 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:54:19,528 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,529 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,529 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,529 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:19,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,531 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,531 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:54:19,536 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-10 13:54:19,536 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:19,536 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:19,536 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-10 13:54:19,536 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,539 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,539 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,541 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,541 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,541 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,543 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,543 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,544 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,544 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:19,543 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,543 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,544 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:19,544 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,645 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:19,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:19,660 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-10 13:54:19,662 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,662 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,662 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-10 13:54:19,662 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:54:19,663 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,663 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,664 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,664 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,665 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:19,666 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,666 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,667 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. started... 2023-02-10 13:54:19,667 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,670 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,670 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,670 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,670 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,670 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,679 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d 2023-02-10 13:54:19,690 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/d59b71787f864d8fa119a9a29b77b195 2023-02-10 13:54:19,694 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d 2023-02-10 13:54:19,698 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:19,698 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/d59b71787f864d8fa119a9a29b77b195 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 2023-02-10 13:54:19,702 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:19,703 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 36ms, sequenceid=58, compaction requested=true 2023-02-10 13:54:19,703 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:19,703 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,705 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,705 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-10 13:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78923c7d[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:19,707 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:54:19,707 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,708 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,708 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,711 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,715 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,715 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,715 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:54:19,717 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-10 13:54:19,717 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:19,717 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:19,717 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-10 13:54:19,717 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,720 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,720 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,720 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,723 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,723 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,724 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,730 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,730 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,730 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,730 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,731 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:19,731 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:19,831 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:19,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:19,847 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:19,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:19,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-10 13:54:19,849 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-10 13:54:19,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:54:19,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,852 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,852 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,852 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,852 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,853 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,853 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:19,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:19,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,854 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-10 13:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:54:19,857 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,857 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,857 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:19,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:19,862 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,862 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. started... 2023-02-10 13:54:19,862 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:19,863 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,864 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,864 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,875 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/8f447d43d779420f8c1c0f64a575a906 2023-02-10 13:54:19,892 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/57f1bf84b08244219232fe2815c11da9 2023-02-10 13:54:19,897 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/8f447d43d779420f8c1c0f64a575a906 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 2023-02-10 13:54:19,902 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:19,903 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/57f1bf84b08244219232fe2815c11da9 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 2023-02-10 13:54:19,908 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:19,909 INFO [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 47ms, sequenceid=71, compaction requested=true 2023-02-10 13:54:19,909 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:19,909 DEBUG [rs(jenkins-hbase4.apache.org,46307,1676037243975)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-10 13:54:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-10 13:54:19,911 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,911 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-10 13:54:19,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c184bb4[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:19,913 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:54:19,913 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,921 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,921 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:19,923 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,923 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,923 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,923 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-10 13:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-10 13:54:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,46307,1676037243975': 2023-02-10 13:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:54:19,929 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-10 13:54:19,929 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:19,929 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:19,929 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-10 13:54:19,929 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,931 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,932 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:19,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,933 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,933 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:19,936 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-10 13:54:19,936 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,948 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,948 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:19,948 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-10 13:54:19,949 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,949 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,949 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,949 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,949 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-10 13:54:19,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:19,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:19,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:19,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-10 13:54:19,950 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-10 13:54:19,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:19,949 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:19,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:20,050 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:20,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:20,312 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-02-10 13:54:20,313 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:20,314 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,315 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,315 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,315 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,316 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:20,316 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:20,316 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:20,564 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,564 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:20,564 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:20,564 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:20,565 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-10 13:54:20,565 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:20,565 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:20,565 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:20,565 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:20,566 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:20,567 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-02-10 13:54:20,567 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:20,567 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:20,568 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:20,568 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:20,568 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:20,568 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:20,568 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:20,569 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:20,569 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:20,569 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:20,569 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:20,569 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:20,570 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:20,570 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,570 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:20,570 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,570 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,570 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,570 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:20,571 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:20,571 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:20,572 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037290572 2023-02-10 13:54:20,573 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:20,573 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,575 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:20,576 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:20,576 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:20,576 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037320576 2023-02-10 13:54:20,576 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037380576 2023-02-10 13:54:20,576 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:20,576 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:20,578 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:20,578 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:20,578 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:20,826 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,826 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:20,826 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:20,827 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:20,828 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:20,828 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:20,828 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:20,828 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:20,828 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:20,829 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:20,829 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-10 13:54:20,830 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:20,830 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:20,830 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:20,830 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:20,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:20,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:20,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:20,832 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:20,832 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:20,832 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:20,833 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:20,833 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:20,833 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:20,833 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,833 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:20,833 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,833 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,834 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,834 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:20,834 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:20,835 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:20,836 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037290836 2023-02-10 13:54:20,837 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:20,837 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,840 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:20,840 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:20,840 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:20,840 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037320840 2023-02-10 13:54:20,840 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037380840 2023-02-10 13:54:20,840 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:20,840 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:20,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN}] 2023-02-10 13:54:20,843 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:20,843 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:20,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:20,843 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:21,091 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:21,091 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:21,092 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:21,093 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-10 13:54:21,093 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:21,094 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:21,094 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:21,094 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:21,094 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,095 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:21,095 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,095 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,095 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,096 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:21,096 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,096 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,097 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:21,097 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,097 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,097 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,097 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,097 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,097 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:21,098 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:21,098 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:21,099 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037291099 2023-02-10 13:54:21,100 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:21,100 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,102 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:21,103 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:21,103 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:21,103 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037321103 2023-02-10 13:54:21,103 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037381103 2023-02-10 13:54:21,103 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:21,103 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN}] 2023-02-10 13:54:21,103 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:21,105 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,106 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,106 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037261106"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037261106"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037261106"}]},"ts":"1676037261106"} 2023-02-10 13:54:21,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:21,108 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:21,108 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:21,108 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:21,353 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,353 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:21,353 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:21,353 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:21,356 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:21,356 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:21,356 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:21,356 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:21,356 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:21,357 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:21,358 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:21,358 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-10 13:54:21,358 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:21,358 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:21,358 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:21,358 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:21,359 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,359 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:21,359 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,359 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,360 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,360 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:21,360 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,360 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,360 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:21,361 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,361 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,361 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,361 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,361 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,361 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,361 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,362 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,362 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:21,362 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:21,363 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:21,363 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:21,364 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037291364 2023-02-10 13:54:21,364 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:21,365 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,367 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:21,367 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSING, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:21,367 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:21,367 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037321367 2023-02-10 13:54:21,367 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037381367 2023-02-10 13:54:21,367 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:21,368 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,368 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037261367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037261367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037261367"}]},"ts":"1676037261367"} 2023-02-10 13:54:21,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:21,370 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:21,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:21,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9223b687f29223ce102064f5414860f7, disabling compactions & flushes 2023-02-10 13:54:21,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:21,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:21,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. after waiting 0 ms 2023-02-10 13:54:21,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:21,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9223b687f29223ce102064f5414860f7 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:54:21,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af 2023-02-10 13:54:21,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/69d84859d9ce43709f6aa24dbb4e38bb 2023-02-10 13:54:21,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af 2023-02-10 13:54:21,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:54:21,555 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/69d84859d9ce43709f6aa24dbb4e38bb as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb 2023-02-10 13:54:21,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:54:21,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 36ms, sequenceid=82, compaction requested=true 2023-02-10 13:54:21,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:54:21,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:21,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:21,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9223b687f29223ce102064f5414860f7: Waiting for close lock at 1676037261525Running coprocessor pre-close hooks at 1676037261525Disabling compacts and flushes for region at 1676037261525Disabling writes for close at 1676037261525Obtaining lock to block concurrent updates at 1676037261525Preparing flush snapshotting stores in 9223b687f29223ce102064f5414860f7 at 1676037261525Finished memstore snapshotting testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1676037261525Flushing stores of testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. at 1676037261526 (+1 ms)Flushing 9223b687f29223ce102064f5414860f7/cf1: creating writer at 1676037261526Flushing 9223b687f29223ce102064f5414860f7/cf1: appending metadata at 1676037261528 (+2 ms)Flushing 9223b687f29223ce102064f5414860f7/cf1: closing flushed file at 1676037261528Flushing 9223b687f29223ce102064f5414860f7/cf2: creating writer at 1676037261539 (+11 ms)Flushing 9223b687f29223ce102064f5414860f7/cf2: appending metadata at 1676037261541 (+2 ms)Flushing 9223b687f29223ce102064f5414860f7/cf2: closing flushed file at 1676037261541Flushing 9223b687f29223ce102064f5414860f7/cf1: reopening flushed file at 1676037261551 (+10 ms)Flushing 9223b687f29223ce102064f5414860f7/cf2: reopening flushed file at 1676037261556 (+5 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 9223b687f29223ce102064f5414860f7 in 36ms, sequenceid=82, compaction requested=true at 1676037261561 (+5 ms)Writing region close event to WAL at 1676037261562 (+1 ms)Running coprocessor post-close hooks at 1676037261565 (+3 ms)Closed at 1676037261565 2023-02-10 13:54:21,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:21,568 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=CLOSED 2023-02-10 13:54:21,568 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037261568"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037261568"}]},"ts":"1676037261568"} 2023-02-10 13:54:21,569 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:21,569 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,569 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:21,569 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:21,617 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,618 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:21,618 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:21,618 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:21,619 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:21,619 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:21,619 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:21,619 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:21,619 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:21,619 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:21,620 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:21,620 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-10 13:54:21,620 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-02-10 13:54:21,621 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:21,621 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:21,621 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:21,621 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:21,622 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,622 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:21,622 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,622 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,622 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,623 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:21,623 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,623 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,623 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:21,624 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,624 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,624 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,624 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,624 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,624 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,624 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,624 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,624 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,624 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,624 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:21,625 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:21,625 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:21,626 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:21,626 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037291626 2023-02-10 13:54:21,627 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:21,627 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,629 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:21,629 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:21,629 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:21,629 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037321629 2023-02-10 13:54:21,629 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037381629 2023-02-10 13:54:21,629 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:21,630 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=CLOSED, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,630 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037261629"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037261629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037261629"}]},"ts":"1676037261629"} 2023-02-10 13:54:21,631 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:21,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=45 2023-02-10 13:54:21,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 in 261 msec 2023-02-10 13:54:21,634 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:21,634 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:21,634 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:21,880 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,880 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:21,880 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:21,880 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:21,881 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:21,881 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:21,881 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:21,881 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:21,881 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:21,882 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:21,883 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:21,883 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-10 13:54:21,883 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:21,883 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:21,884 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:21,884 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:21,884 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:21,884 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,885 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:21,885 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,885 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:21,885 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,886 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:21,886 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,886 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:21,886 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:21,887 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,887 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,887 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:21,887 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:21,887 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,887 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,887 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,887 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:21,887 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:21,887 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:21,888 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:21,889 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:21,889 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:21,889 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037291889 2023-02-10 13:54:21,889 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:21,890 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,892 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:21,892 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:21,892 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:21,892 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037321892 2023-02-10 13:54:21,892 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037381892 2023-02-10 13:54:21,892 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:21,893 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:21,894 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:21,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-02-10 13:54:21,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN in 791 msec 2023-02-10 13:54:21,899 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:21,899 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:21,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:21,900 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:22,143 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,143 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:22,143 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:22,143 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:22,144 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:22,144 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:22,144 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:22,144 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:22,144 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:22,145 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:22,146 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:22,146 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:22,146 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:22,146 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:22,146 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:22,147 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:22,147 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:22,147 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:22,147 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:22,147 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:22,148 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:22,148 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:22,148 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:22,148 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:22,148 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:22,149 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:22,149 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:22,150 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:22,150 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:22,160 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037292160 2023-02-10 13:54:22,161 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,161 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:22,163 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:22,164 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:22,164 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:22,164 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037322164 2023-02-10 13:54:22,164 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037382164 2023-02-10 13:54:22,164 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:22,164 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,164 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,164 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,169 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:22,178 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=9223b687f29223ce102064f5414860f7, threads=14 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,179 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,180 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,180 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,180 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,184 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,184 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,184 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,184 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,389 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,391 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,403 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,413 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,416 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,421 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,423 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,438 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,441 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,442 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,442 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,442 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,443 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,444 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,444 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 9223b687f29223ce102064f5414860f7 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:54:22,480 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:22,480 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,480 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:22,480 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:22,661 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:54:22,664 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,664 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:22,664 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:22,665 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:22,665 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:22,665 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:22,665 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:22,665 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:22,665 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:22,666 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:22,667 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-10 13:54:22,667 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:22,667 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:22,667 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:22,667 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:22,668 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:22,668 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:22,668 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:22,668 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:22,668 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:22,668 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:22,669 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:22,669 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:22,669 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:22,669 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:22,670 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,670 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,670 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:22,670 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:22,671 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:22,671 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:22,672 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:22,673 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037292673 2023-02-10 13:54:22,674 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:22,674 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,677 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:22,677 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:22,677 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:22,677 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037322677 2023-02-10 13:54:22,678 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037382678 2023-02-10 13:54:22,678 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:22,690 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=9223b687f29223ce102064f5414860f7, threads=14 2023-02-10 13:54:22,693 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,693 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,700 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,700 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,702 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,704 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,704 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,705 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,710 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,713 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,716 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,720 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,720 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,721 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,816 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,863 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,891 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,909 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,910 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,912 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,927 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,936 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,936 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,938 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,939 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,944 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,944 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,948 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb for region: 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:22,948 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 9223b687f29223ce102064f5414860f7 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:54:22,988 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:22,993 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/61b0236e59d564d7f55a2300cc57efa2/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:54:22,995 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/c09ff3b9bd6b60cc75155fde202ce9ac/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:54:22,995 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:22,995 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:22,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:22,995 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:23,178 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,178 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:23,178 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:23,178 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:23,179 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:23,180 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:23,180 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:23,180 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:23,180 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:23,181 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:23,182 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-10 13:54:23,182 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:23,182 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:23,182 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:23,183 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:23,183 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:23,183 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:23,184 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,184 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:23,184 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,184 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,185 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,185 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:23,185 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,185 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,185 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:23,186 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,186 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:23,186 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:23,188 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:23,188 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,188 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,188 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,188 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,188 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,188 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-02-10 13:54:23,189 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:23,190 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:23,190 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037293190 2023-02-10 13:54:23,192 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:23,192 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,194 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:23,194 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:23,195 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:23,195 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037323195 2023-02-10 13:54:23,195 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037383195 2023-02-10 13:54:23,195 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:23,197 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:23,200 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:23,200 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,200 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:23,200 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:23,445 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 0 rollback state: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,446 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:23,446 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:23,446 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:23,447 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:23,447 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:23,447 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:23,447 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:23,447 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:23,448 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:23,449 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-10 13:54:23,449 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:23,449 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:23,449 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:23,449 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:23,449 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:23,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:23,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:23,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,450 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:23,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,451 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:23,452 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,452 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,452 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,452 DEBUG [Listener at localhost/43477] procedure2.StateMachineProcedure(225): Abort requested for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,453 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:23,453 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:23,454 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:23,454 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037293454 2023-02-10 13:54:23,454 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:23,457 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,458 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:23,459 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:23,459 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:23,459 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037323459 2023-02-10 13:54:23,459 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037383459 2023-02-10 13:54:23,459 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-10 13:54:23,459 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:23,472 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:23,472 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:23,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:23,472 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:23,710 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 1 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,710 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:23,710 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:23,710 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:23,711 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:23,711 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:23,711 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:23,712 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:23,712 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:23,712 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:23,713 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-02-10 13:54:23,713 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:23,714 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:23,714 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:23,714 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:23,715 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:23,715 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:23,715 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,715 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:23,716 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,716 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,716 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,716 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:23,716 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,716 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,717 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:23,718 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,718 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:23,718 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:23,718 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:23,718 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,718 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,719 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,719 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,719 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,719 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-10 13:54:23,719 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:23,720 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:23,720 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037293720 2023-02-10 13:54:23,723 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:23,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,725 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:23,725 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:23,725 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:23,725 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037323725 2023-02-10 13:54:23,725 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037383725 2023-02-10 13:54:23,725 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:23,726 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:23,728 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:23,728 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:23,728 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:23,728 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:23,977 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 2 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,977 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:23,977 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:23,977 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:23,977 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:23,978 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:23,978 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:23,978 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:23,978 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:23,978 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:23,979 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-10 13:54:23,979 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:23,979 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:23,980 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:23,980 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:23,980 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:23,980 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:23,981 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,981 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:23,981 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,981 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:23,981 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,982 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:23,982 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,982 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:23,982 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:23,983 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,983 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:23,983 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:23,983 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:23,983 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,983 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,983 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,983 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,983 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:23,983 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:23,984 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:23,984 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:23,985 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037293985 2023-02-10 13:54:23,985 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:23,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:23,988 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:23,988 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:23,989 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:23,989 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037323989 2023-02-10 13:54:23,989 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037383989 2023-02-10 13:54:23,989 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:23,989 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:23,991 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:23,991 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:23,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:23,991 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:24,239 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 3 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,239 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:24,239 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:24,239 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:24,239 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:24,240 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:24,240 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:24,240 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:24,240 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:24,240 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:24,241 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-10 13:54:24,241 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:24,241 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:24,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:24,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:24,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:24,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:24,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:24,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:24,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:24,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:24,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:24,245 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:24,245 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,245 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,245 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,245 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:24,245 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:24,245 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:24,246 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:24,246 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:24,246 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037294246 2023-02-10 13:54:24,248 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:24,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,250 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:24,250 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:24,250 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:24,250 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037324250 2023-02-10 13:54:24,251 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037384251 2023-02-10 13:54:24,251 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:54:24,251 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:24,254 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1054): Stored [pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN] 2023-02-10 13:54:24,254 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:24,254 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:24,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:24,254 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:24,501 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 4 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,501 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:24,501 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:24,501 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:24,502 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:24,502 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:24,503 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:24,503 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:24,503 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:24,503 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:24,504 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-10 13:54:24,504 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:54:24,504 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:54:24,504 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-10 13:54:24,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:24,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:24,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:24,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:24,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:24,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:24,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:24,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:24,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:24,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:24,508 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:24,509 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,509 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,509 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,509 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:24,509 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:24,509 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:24,509 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:24,510 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:24,510 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:24,510 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:24,511 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037294511 2023-02-10 13:54:24,512 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:24,513 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,514 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:24,516 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:24,516 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:24,516 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:24,516 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037324516 2023-02-10 13:54:24,516 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037384516 2023-02-10 13:54:24,516 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:24,516 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:24,522 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:24,522 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:24,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:24,522 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:24,766 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 5 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,767 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:24,767 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:24,767 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:24,769 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:24,769 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:24,769 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:24,769 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:24,769 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:24,770 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:24,771 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:54:24,771 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-10 13:54:24,771 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:24,771 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:24,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:24,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:24,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:24,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:24,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:24,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:24,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:24,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:24,774 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:24,774 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,774 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,774 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,774 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:24,774 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:24,775 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:24,775 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:24,775 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:24,776 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037294776 2023-02-10 13:54:24,776 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:24,778 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:24,779 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:24,780 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:24,780 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:24,781 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:24,781 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037324781 2023-02-10 13:54:24,781 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037384781 2023-02-10 13:54:24,781 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-10 13:54:24,781 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:24,783 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:24,783 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:24,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:24,783 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:25,031 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 6 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,031 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:25,031 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:25,031 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:25,031 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:25,031 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:25,032 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:25,032 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:25,032 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:25,032 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:25,033 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-10 13:54:25,033 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-10 13:54:25,033 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:25,034 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:25,034 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:25,034 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:25,034 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,034 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:25,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:25,036 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,036 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,036 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:25,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,037 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:25,037 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,037 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,037 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,038 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:25,038 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:25,038 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:25,039 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:25,039 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:25,039 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037295039 2023-02-10 13:54:25,039 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:25,041 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,042 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,044 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:25,044 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:25,044 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:25,044 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037325044 2023-02-10 13:54:25,045 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037385045 2023-02-10 13:54:25,045 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-10 13:54:25,045 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:25,047 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:25,047 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-10 13:54:25,047 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:25,047 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(509): Restart 7 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:25,295 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:25,295 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:25,296 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:25,296 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:25,297 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-10 13:54:25,297 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-10 13:54:25,297 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:25,298 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:25,298 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:25,298 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:25,298 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,299 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:25,299 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,299 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,299 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,300 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:25,300 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,300 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,300 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:25,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,301 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:25,301 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,301 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,302 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,302 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:54:25,302 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-10 13:54:25,302 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:25,303 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:25,303 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:25,303 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037295303 2023-02-10 13:54:25,304 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:25,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,308 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,310 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:25,310 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:25,310 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:25,310 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037325310 2023-02-10 13:54:25,310 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037385310 2023-02-10 13:54:25,310 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-10 13:54:25,310 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:54:25,313 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac exec-time=4.9970 sec 2023-02-10 13:54:25,313 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,313 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=true 2023-02-10 13:54:25,464 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:25,465 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:25,465 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037265465"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037265465"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037265465"}]},"ts":"1676037265465"} 2023-02-10 13:54:25,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:25,467 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:54:25,467 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:54:25,468 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:54:25,560 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:25,560 WARN [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(526): ProcedureExecutor not running, may have been stopped by pending procedure due to KillAndToggleBeforeStoreUpdate flag. 2023-02-10 13:54:25,560 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:54:25,561 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:54:25,561 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:54:25,561 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:54:25,561 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:54:25,561 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:54:25,561 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:54:25,561 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:54:25,562 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:54:25,563 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[0] 2023-02-10 13:54:25,563 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:54:25,563 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:54:25,563 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:54:25,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:54:25,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:54:25,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,564 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:54:25,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:54:25,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,565 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:54:25,566 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:25,566 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:54:25,566 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,566 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:54:25,566 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,567 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-10 13:54:25,567 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:54:25,568 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:54:25,568 INFO [Listener at localhost/43477] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7 to restore RIT 2023-02-10 13:54:25,568 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037295568 2023-02-10 13:54:25,568 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:54:25,569 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:54:25,571 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:54:25,571 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=9223b687f29223ce102064f5414860f7, regionState=OPENING, lastHost=jenkins-hbase4.apache.org,46307,1676037243975, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975, openSeqNum=2 2023-02-10 13:54:25,571 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:54:25,571 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037325571 2023-02-10 13:54:25,571 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037385571 2023-02-10 13:54:25,571 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:54:25,572 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:25,572 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037265572"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037265572"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037265572"}]},"ts":"1676037265572"} 2023-02-10 13:54:25,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:25,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:25,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9223b687f29223ce102064f5414860f7, NAME => 'testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:25,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:25,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:25,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,730 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,731 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1 2023-02-10 13:54:25,731 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1 2023-02-10 13:54:25,731 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9223b687f29223ce102064f5414860f7 columnFamilyName cf1 2023-02-10 13:54:25,737 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d 2023-02-10 13:54:25,740 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 2023-02-10 13:54:25,743 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b 2023-02-10 13:54:25,746 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f 2023-02-10 13:54:25,749 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af 2023-02-10 13:54:25,752 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 2023-02-10 13:54:25,755 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 2023-02-10 13:54:25,755 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(310): Store=9223b687f29223ce102064f5414860f7/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:25,755 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,756 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2 2023-02-10 13:54:25,756 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2 2023-02-10 13:54:25,757 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9223b687f29223ce102064f5414860f7 columnFamilyName cf2 2023-02-10 13:54:25,762 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 2023-02-10 13:54:25,765 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb 2023-02-10 13:54:25,768 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 2023-02-10 13:54:25,771 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb 2023-02-10 13:54:25,774 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 2023-02-10 13:54:25,778 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 2023-02-10 13:54:25,781 DEBUG [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 2023-02-10 13:54:25,781 INFO [StoreOpener-9223b687f29223ce102064f5414860f7-1] regionserver.HStore(310): Store=9223b687f29223ce102064f5414860f7/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:25,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(602): Cleaned up old failed split transaction detritus: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.splits 2023-02-10 13:54:25,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:25,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:25,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9223b687f29223ce102064f5414860f7; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69170387, jitterRate=0.030719086527824402}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:25,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9223b687f29223ce102064f5414860f7: Running coprocessor pre-open hook at 1676037265729Writing region info on filesystem at 1676037265729Initializing all the Stores at 1676037265730 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037265730Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037265730Cleaning up temporary data from old regions at 1676037265783 (+53 ms)Cleaning up detritus from prior splits at 1676037265783Running coprocessor post-open hooks at 1676037265787 (+4 ms)Region opened successfully at 1676037265787 2023-02-10 13:54:25,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7., pid=48, masterSystemTime=1676037265726 2023-02-10 13:54:25,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:25,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:25,789 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:25,789 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:25,789 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-10 13:54:25,789 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-10 13:54:25,790 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1912): 9223b687f29223ce102064f5414860f7/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:25,790 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1912): 9223b687f29223ce102064f5414860f7/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:25,790 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 9223b687f29223ce102064f5414860f7/cf2 in testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:25,790 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 9223b687f29223ce102064f5414860f7/cf1 in testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:25,790 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp, totalSize=34.9 K 2023-02-10 13:54:25,790 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp, totalSize=34.9 K 2023-02-10 13:54:25,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:25,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:25,791 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 1a3e690bb1014153bc2ccef997bd7e98, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037258910 2023-02-10 13:54:25,791 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 39d92a278982404a8fabb4e6fad7907f, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037258910 2023-02-10 13:54:25,791 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:25,792 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting fa7e61d1f1f6472caa52d191eb388dd2, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037259091 2023-02-10 13:54:25,792 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037265791"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037265791"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037265791"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037265791"}]},"ts":"1676037265791"} 2023-02-10 13:54:25,792 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 32772e7cd7a74abea498a01369f95e5b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037259091 2023-02-10 13:54:25,792 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting d121748ce713426d969589a390a13ce3, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037259282 2023-02-10 13:54:25,792 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting fbb4f2f534f1495bad519f870de9d2c6, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037259282 2023-02-10 13:54:25,792 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 518bfa8b4be84d3984404c2670636ebb, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037259468 2023-02-10 13:54:25,793 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 1488f2ad74af485caccfd05315c5be90, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037259468 2023-02-10 13:54:25,793 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting d59b71787f864d8fa119a9a29b77b195, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037259646 2023-02-10 13:54:25,793 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 11fb50e49d114d1ea7ac0ba5ba94dc6d, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037259646 2023-02-10 13:54:25,793 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 57f1bf84b08244219232fe2815c11da9, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037259833 2023-02-10 13:54:25,794 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 8f447d43d779420f8c1c0f64a575a906, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037259833 2023-02-10 13:54:25,794 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] compactions.Compactor(207): Compacting 69d84859d9ce43709f6aa24dbb4e38bb, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037260052 2023-02-10 13:54:25,794 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] compactions.Compactor(207): Compacting 56dc7221e82f4dc5b50b2fa8e7abd8af, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037260052 2023-02-10 13:54:25,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-02-10 13:54:25,796 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; OpenRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 in 220 msec 2023-02-10 13:54:25,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN in 1.5430 sec 2023-02-10 13:54:25,817 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9223b687f29223ce102064f5414860f7#cf1#compaction#51 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:25,817 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] throttle.PressureAwareThroughputController(145): 9223b687f29223ce102064f5414860f7#cf2#compaction#50 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:25,822 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(335): procId=44 exception: org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested 2023-02-10 13:54:25,825 DEBUG [Listener at localhost/43477] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRollbackAndDoubleExecution get assigned. Timeout = 1500ms 2023-02-10 13:54:25,827 INFO [Listener at localhost/43477] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-02-10 13:54:25,842 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf1/6561695e6e9542aab76a35a516f265c0 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/6561695e6e9542aab76a35a516f265c0 2023-02-10 13:54:25,847 INFO [Listener at localhost/43477] hbase.HBaseTestingUtility(3484): All regions for table testRollbackAndDoubleExecution assigned to meta. Checking AM states. 2023-02-10 13:54:25,847 INFO [Listener at localhost/43477] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-02-10 13:54:25,847 INFO [Listener at localhost/43477] hbase.HBaseTestingUtility(3504): All regions for table testRollbackAndDoubleExecution assigned. 2023-02-10 13:54:25,852 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9223b687f29223ce102064f5414860f7/cf1 of 9223b687f29223ce102064f5414860f7 into 6561695e6e9542aab76a35a516f265c0(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:25,852 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:25,853 INFO [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7., storeName=9223b687f29223ce102064f5414860f7/cf1, priority=9, startTime=1676037265788; duration=0sec 2023-02-10 13:54:25,853 DEBUG [RS:1;jenkins-hbase4:46307-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:25,854 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/.tmp/cf2/f24d985b5eb44722bb9f812c16330634 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/f24d985b5eb44722bb9f812c16330634 2023-02-10 13:54:25,861 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9223b687f29223ce102064f5414860f7/cf2 of 9223b687f29223ce102064f5414860f7 into f24d985b5eb44722bb9f812c16330634(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:25,861 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 9223b687f29223ce102064f5414860f7: 2023-02-10 13:54:25,862 INFO [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7., storeName=9223b687f29223ce102064f5414860f7/cf2, priority=9, startTime=1676037265789; duration=0sec 2023-02-10 13:54:25,862 DEBUG [RS:1;jenkins-hbase4:46307-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:25,872 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:25,874 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testRollbackAndDoubleExecution 2023-02-10 13:54:25,875 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRollbackAndDoubleExecution 2023-02-10 13:54:25,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:25,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-10 13:54:25,878 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037265878"}]},"ts":"1676037265878"} 2023-02-10 13:54:25,879 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLING in hbase:meta 2023-02-10 13:54:25,881 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackAndDoubleExecution to state=DISABLING 2023-02-10 13:54:25,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN}] 2023-02-10 13:54:25,883 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN 2023-02-10 13:54:25,883 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:25,883 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037265883"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037265883"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037265883"}]},"ts":"1676037265883"} 2023-02-10 13:54:25,885 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:25,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-10 13:54:26,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:26,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9223b687f29223ce102064f5414860f7, disabling compactions & flushes 2023-02-10 13:54:26,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:26,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:26,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. after waiting 0 ms 2023-02-10 13:54:26,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:26,037 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af] to archive 2023-02-10 13:54:26,038 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:26,040 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/39d92a278982404a8fabb4e6fad7907f 2023-02-10 13:54:26,041 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/32772e7cd7a74abea498a01369f95e5b 2023-02-10 13:54:26,043 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/fbb4f2f534f1495bad519f870de9d2c6 2023-02-10 13:54:26,044 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/1488f2ad74af485caccfd05315c5be90 2023-02-10 13:54:26,045 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/11fb50e49d114d1ea7ac0ba5ba94dc6d 2023-02-10 13:54:26,046 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/8f447d43d779420f8c1c0f64a575a906 2023-02-10 13:54:26,048 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/56dc7221e82f4dc5b50b2fa8e7abd8af 2023-02-10 13:54:26,053 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb] to archive 2023-02-10 13:54:26,053 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:26,055 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/1a3e690bb1014153bc2ccef997bd7e98 2023-02-10 13:54:26,056 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/fa7e61d1f1f6472caa52d191eb388dd2 2023-02-10 13:54:26,057 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d121748ce713426d969589a390a13ce3 2023-02-10 13:54:26,058 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/518bfa8b4be84d3984404c2670636ebb 2023-02-10 13:54:26,059 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/d59b71787f864d8fa119a9a29b77b195 2023-02-10 13:54:26,060 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/57f1bf84b08244219232fe2815c11da9 2023-02-10 13:54:26,062 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/69d84859d9ce43709f6aa24dbb4e38bb 2023-02-10 13:54:26,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:54:26,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:26,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7. 2023-02-10 13:54:26,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9223b687f29223ce102064f5414860f7: Waiting for close lock at 1676037266037Running coprocessor pre-close hooks at 1676037266037Disabling compacts and flushes for region at 1676037266037Disabling writes for close at 1676037266037Writing region close event to WAL at 1676037266063 (+26 ms)Running coprocessor post-close hooks at 1676037266066 (+3 ms)Closed at 1676037266066 2023-02-10 13:54:26,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9223b687f29223ce102064f5414860f7 2023-02-10 13:54:26,068 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=9223b687f29223ce102064f5414860f7, regionState=CLOSED 2023-02-10 13:54:26,068 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037266068"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037266068"}]},"ts":"1676037266068"} 2023-02-10 13:54:26,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-02-10 13:54:26,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; CloseRegionProcedure 9223b687f29223ce102064f5414860f7, server=jenkins-hbase4.apache.org,46307,1676037243975 in 184 msec 2023-02-10 13:54:26,073 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-02-10 13:54:26,073 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, UNASSIGN in 189 msec 2023-02-10 13:54:26,073 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037266073"}]},"ts":"1676037266073"} 2023-02-10 13:54:26,074 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLED in hbase:meta 2023-02-10 13:54:26,077 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackAndDoubleExecution to state=DISABLED 2023-02-10 13:54:26,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution in 202 msec 2023-02-10 13:54:26,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-10 13:54:26,180 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackAndDoubleExecution, procId: 49 completed 2023-02-10 13:54:26,181 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRollbackAndDoubleExecution 2023-02-10 13:54:26,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:26,183 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:26,184 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=52, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:26,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-02-10 13:54:26,188 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:26,190 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/recovered.edits] 2023-02-10 13:54:26,192 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/6561695e6e9542aab76a35a516f265c0 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf1/6561695e6e9542aab76a35a516f265c0 2023-02-10 13:54:26,195 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/f24d985b5eb44722bb9f812c16330634 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/cf2/f24d985b5eb44722bb9f812c16330634 2023-02-10 13:54:26,197 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7/recovered.edits/90.seqid 2023-02-10 13:54:26,198 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackAndDoubleExecution/9223b687f29223ce102064f5414860f7 2023-02-10 13:54:26,198 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-02-10 13:54:26,201 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=52, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:26,203 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackAndDoubleExecution from hbase:meta 2023-02-10 13:54:26,204 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackAndDoubleExecution' descriptor. 2023-02-10 13:54:26,205 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=52, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:26,205 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackAndDoubleExecution' from region states. 2023-02-10 13:54:26,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037266205"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:26,207 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-02-10 13:54:26,207 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9223b687f29223ce102064f5414860f7, NAME => 'testRollbackAndDoubleExecution,,1676037258406.9223b687f29223ce102064f5414860f7.', STARTKEY => '', ENDKEY => ''}] 2023-02-10 13:54:26,207 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackAndDoubleExecution' as deleted. 2023-02-10 13:54:26,207 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037266207"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:26,208 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackAndDoubleExecution state from META 2023-02-10 13:54:26,210 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=52, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:54:26,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution in 29 msec 2023-02-10 13:54:26,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-02-10 13:54:26,286 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackAndDoubleExecution, procId: 52 completed 2023-02-10 13:54:26,300 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=399 (was 403), OpenFileDescriptor=866 (was 871), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=227 (was 270), ProcessCount=168 (was 168), AvailableMemoryMB=3439 (was 3522) 2023-02-10 13:54:26,313 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=399, OpenFileDescriptor=866, MaxFileDescriptor=60000, SystemLoadAverage=227, ProcessCount=168, AvailableMemoryMB=3438 2023-02-10 13:54:26,313 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:26,314 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:54:26,315 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:54:26,316 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:26,317 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:26,318 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:26,320 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,320 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 empty. 2023-02-10 13:54:26,321 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,321 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-02-10 13:54:26,333 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:26,334 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(7671): creating {ENCODED => 0cb6d3ddcccf2a26445c39a07e67b5a6, NAME => 'testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionNoStoreFile', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:26,355 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:26,355 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1603): Closing 0cb6d3ddcccf2a26445c39a07e67b5a6, disabling compactions & flushes 2023-02-10 13:54:26,355 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,355 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,355 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. after waiting 0 ms 2023-02-10 13:54:26,355 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,355 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,355 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1557): Region close journal for 0cb6d3ddcccf2a26445c39a07e67b5a6: Waiting for close lock at 1676037266355Disabling compacts and flushes for region at 1676037266355Disabling writes for close at 1676037266355Writing region close event to WAL at 1676037266355Closed at 1676037266355 2023-02-10 13:54:26,358 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:26,358 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1676037266358"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037266358"}]},"ts":"1676037266358"} 2023-02-10 13:54:26,360 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:26,360 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:26,361 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037266361"}]},"ts":"1676037266361"} 2023-02-10 13:54:26,362 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLING in hbase:meta 2023-02-10 13:54:26,366 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:26,367 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:26,367 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:26,367 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:26,367 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:26,367 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, ASSIGN}] 2023-02-10 13:54:26,368 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, ASSIGN 2023-02-10 13:54:26,369 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:26,519 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:26,520 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=0cb6d3ddcccf2a26445c39a07e67b5a6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:26,520 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1676037266520"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037266520"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037266520"}]},"ts":"1676037266520"} 2023-02-10 13:54:26,522 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE; OpenRegionProcedure 0cb6d3ddcccf2a26445c39a07e67b5a6, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:26,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0cb6d3ddcccf2a26445c39a07e67b5a6, NAME => 'testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:26,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:26,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:26,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,679 INFO [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,680 DEBUG [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/cf1 2023-02-10 13:54:26,680 DEBUG [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/cf1 2023-02-10 13:54:26,681 INFO [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0cb6d3ddcccf2a26445c39a07e67b5a6 columnFamilyName cf1 2023-02-10 13:54:26,681 INFO [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] regionserver.HStore(310): Store=0cb6d3ddcccf2a26445c39a07e67b5a6/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:26,681 INFO [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,683 DEBUG [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/cf2 2023-02-10 13:54:26,683 DEBUG [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/cf2 2023-02-10 13:54:26,683 INFO [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0cb6d3ddcccf2a26445c39a07e67b5a6 columnFamilyName cf2 2023-02-10 13:54:26,684 INFO [StoreOpener-0cb6d3ddcccf2a26445c39a07e67b5a6-1] regionserver.HStore(310): Store=0cb6d3ddcccf2a26445c39a07e67b5a6/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:26,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:26,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:26,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 0cb6d3ddcccf2a26445c39a07e67b5a6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73540382, jitterRate=0.09583708643913269}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:26,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 0cb6d3ddcccf2a26445c39a07e67b5a6: Running coprocessor pre-open hook at 1676037266678Writing region info on filesystem at 1676037266678Initializing all the Stores at 1676037266678Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037266678Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037266679 (+1 ms)Cleaning up temporary data from old regions at 1676037266686 (+7 ms)Cleaning up detritus from prior splits at 1676037266686Running coprocessor post-open hooks at 1676037266690 (+4 ms)Region opened successfully at 1676037266690 2023-02-10 13:54:26,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6., pid=55, masterSystemTime=1676037266674 2023-02-10 13:54:26,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,693 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=0cb6d3ddcccf2a26445c39a07e67b5a6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:26,693 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1676037266693"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037266693"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037266693"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037266693"}]},"ts":"1676037266693"} 2023-02-10 13:54:26,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-02-10 13:54:26,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; OpenRegionProcedure 0cb6d3ddcccf2a26445c39a07e67b5a6, server=jenkins-hbase4.apache.org,46307,1676037243975 in 172 msec 2023-02-10 13:54:26,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=54, resume processing ppid=53 2023-02-10 13:54:26,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=53, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, ASSIGN in 329 msec 2023-02-10 13:54:26,698 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:26,698 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037266698"}]},"ts":"1676037266698"} 2023-02-10 13:54:26,699 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLED in hbase:meta 2023-02-10 13:54:26,701 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:26,703 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile in 386 msec 2023-02-10 13:54:26,817 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:26,818 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:26,819 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:26,819 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:26,819 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:26,821 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, UNASSIGN}] 2023-02-10 13:54:26,822 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, UNASSIGN 2023-02-10 13:54:26,822 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=0cb6d3ddcccf2a26445c39a07e67b5a6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:26,822 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1676037266822"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037266822"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037266822"}]},"ts":"1676037266822"} 2023-02-10 13:54:26,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE; CloseRegionProcedure 0cb6d3ddcccf2a26445c39a07e67b5a6, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:26,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 0cb6d3ddcccf2a26445c39a07e67b5a6, disabling compactions & flushes 2023-02-10 13:54:26,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. after waiting 0 ms 2023-02-10 13:54:26,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-10 13:54:26,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:26,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6. 2023-02-10 13:54:26,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 0cb6d3ddcccf2a26445c39a07e67b5a6: Waiting for close lock at 1676037266976Running coprocessor pre-close hooks at 1676037266976Disabling compacts and flushes for region at 1676037266976Disabling writes for close at 1676037266977 (+1 ms)Writing region close event to WAL at 1676037266977Running coprocessor post-close hooks at 1676037266981 (+4 ms)Closed at 1676037266981 2023-02-10 13:54:26,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:26,983 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=0cb6d3ddcccf2a26445c39a07e67b5a6, regionState=CLOSED 2023-02-10 13:54:26,983 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1676037266983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037266983"}]},"ts":"1676037266983"} 2023-02-10 13:54:26,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-02-10 13:54:26,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; CloseRegionProcedure 0cb6d3ddcccf2a26445c39a07e67b5a6, server=jenkins-hbase4.apache.org,46307,1676037243975 in 160 msec 2023-02-10 13:54:26,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=56 2023-02-10 13:54:26,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=0cb6d3ddcccf2a26445c39a07e67b5a6, UNASSIGN in 165 msec 2023-02-10 13:54:27,012 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-02-10 13:54:27,014 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-02-10 13:54:27,015 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1676037267015"},{"qualifier":"splitA","vlen":67,"tag":[],"timestamp":"1676037267015"},{"qualifier":"splitB","vlen":67,"tag":[],"timestamp":"1676037267015"}]},"ts":"1676037267015"} 2023-02-10 13:54:27,015 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267015"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037267015"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037267015"}]},"ts":"1676037267015"} 2023-02-10 13:54:27,016 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267015"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037267015"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037267015"}]},"ts":"1676037267015"} 2023-02-10 13:54:27,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, ASSIGN}, {pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, ASSIGN}] 2023-02-10 13:54:27,019 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, ASSIGN 2023-02-10 13:54:27,020 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:27,020 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, ASSIGN 2023-02-10 13:54:27,021 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:27,170 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:54:27,172 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=61e4dbae3501200f091cac3fbd180757, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:27,172 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267172"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037267172"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037267172"}]},"ts":"1676037267172"} 2023-02-10 13:54:27,173 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=60, state=RUNNABLE; OpenRegionProcedure 61e4dbae3501200f091cac3fbd180757, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:27,176 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=68e679423cba5a5772a0f4bdc61f656e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:27,176 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037267176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037267176"}]},"ts":"1676037267176"} 2023-02-10 13:54:27,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=62, ppid=59, state=RUNNABLE; OpenRegionProcedure 68e679423cba5a5772a0f4bdc61f656e, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:27,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61e4dbae3501200f091cac3fbd180757, NAME => 'testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.', STARTKEY => '41', ENDKEY => ''} 2023-02-10 13:54:27,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,330 INFO [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,331 DEBUG [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/cf1 2023-02-10 13:54:27,331 DEBUG [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/cf1 2023-02-10 13:54:27,331 INFO [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61e4dbae3501200f091cac3fbd180757 columnFamilyName cf1 2023-02-10 13:54:27,332 INFO [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] regionserver.HStore(310): Store=61e4dbae3501200f091cac3fbd180757/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:27,332 INFO [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,333 DEBUG [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/cf2 2023-02-10 13:54:27,333 DEBUG [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/cf2 2023-02-10 13:54:27,333 INFO [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61e4dbae3501200f091cac3fbd180757 columnFamilyName cf2 2023-02-10 13:54:27,334 INFO [StoreOpener-61e4dbae3501200f091cac3fbd180757-1] regionserver.HStore(310): Store=61e4dbae3501200f091cac3fbd180757/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:27,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:27,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61e4dbae3501200f091cac3fbd180757; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70203194, jitterRate=0.04610911011695862}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:27,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61e4dbae3501200f091cac3fbd180757: Running coprocessor pre-open hook at 1676037267329Writing region info on filesystem at 1676037267329Initializing all the Stores at 1676037267329Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037267329Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037267329Cleaning up temporary data from old regions at 1676037267336 (+7 ms)Cleaning up detritus from prior splits at 1676037267337 (+1 ms)Running coprocessor post-open hooks at 1676037267339 (+2 ms)Region opened successfully at 1676037267339 2023-02-10 13:54:27,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757., pid=61, masterSystemTime=1676037267325 2023-02-10 13:54:27,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 68e679423cba5a5772a0f4bdc61f656e, NAME => 'testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.', STARTKEY => '', ENDKEY => '41'} 2023-02-10 13:54:27,342 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=61e4dbae3501200f091cac3fbd180757, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:27,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:27,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,342 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267342"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037267342"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037267342"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037267342"}]},"ts":"1676037267342"} 2023-02-10 13:54:27,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:27,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,344 INFO [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,345 DEBUG [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/cf1 2023-02-10 13:54:27,345 DEBUG [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/cf1 2023-02-10 13:54:27,345 INFO [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68e679423cba5a5772a0f4bdc61f656e columnFamilyName cf1 2023-02-10 13:54:27,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=60 2023-02-10 13:54:27,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=60, state=SUCCESS; OpenRegionProcedure 61e4dbae3501200f091cac3fbd180757, server=jenkins-hbase4.apache.org,46307,1676037243975 in 171 msec 2023-02-10 13:54:27,346 INFO [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] regionserver.HStore(310): Store=68e679423cba5a5772a0f4bdc61f656e/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:27,346 INFO [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,347 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, ASSIGN in 328 msec 2023-02-10 13:54:27,347 DEBUG [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/cf2 2023-02-10 13:54:27,347 DEBUG [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/cf2 2023-02-10 13:54:27,348 INFO [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68e679423cba5a5772a0f4bdc61f656e columnFamilyName cf2 2023-02-10 13:54:27,348 INFO [StoreOpener-68e679423cba5a5772a0f4bdc61f656e-1] regionserver.HStore(310): Store=68e679423cba5a5772a0f4bdc61f656e/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:27,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:27,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 68e679423cba5a5772a0f4bdc61f656e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62775083, jitterRate=-0.06457836925983429}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:27,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 68e679423cba5a5772a0f4bdc61f656e: Running coprocessor pre-open hook at 1676037267343Writing region info on filesystem at 1676037267343Initializing all the Stores at 1676037267343Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037267343Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037267343Cleaning up temporary data from old regions at 1676037267350 (+7 ms)Cleaning up detritus from prior splits at 1676037267351 (+1 ms)Running coprocessor post-open hooks at 1676037267353 (+2 ms)Region opened successfully at 1676037267353 2023-02-10 13:54:27,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e., pid=62, masterSystemTime=1676037267325 2023-02-10 13:54:27,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,355 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=68e679423cba5a5772a0f4bdc61f656e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:27,356 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267355"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037267355"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037267355"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037267355"}]},"ts":"1676037267355"} 2023-02-10 13:54:27,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=62, resume processing ppid=59 2023-02-10 13:54:27,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, ppid=59, state=SUCCESS; OpenRegionProcedure 68e679423cba5a5772a0f4bdc61f656e, server=jenkins-hbase4.apache.org,46307,1676037243975 in 180 msec 2023-02-10 13:54:27,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=56 2023-02-10 13:54:27,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, ASSIGN in 340 msec 2023-02-10 13:54:27,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 in 543 msec 2023-02-10 13:54:27,572 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:54:27,575 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:27,576 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,576 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-10 13:54:27,580 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037267580"}]},"ts":"1676037267580"} 2023-02-10 13:54:27,581 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLING in hbase:meta 2023-02-10 13:54:27,582 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionNoStoreFile to state=DISABLING 2023-02-10 13:54:27,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, UNASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, UNASSIGN}] 2023-02-10 13:54:27,584 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, UNASSIGN 2023-02-10 13:54:27,584 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=61e4dbae3501200f091cac3fbd180757, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:27,584 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267584"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037267584"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037267584"}]},"ts":"1676037267584"} 2023-02-10 13:54:27,585 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE; CloseRegionProcedure 61e4dbae3501200f091cac3fbd180757, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:27,587 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, UNASSIGN 2023-02-10 13:54:27,588 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=68e679423cba5a5772a0f4bdc61f656e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:27,588 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267588"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037267588"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037267588"}]},"ts":"1676037267588"} 2023-02-10 13:54:27,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=67, ppid=64, state=RUNNABLE; CloseRegionProcedure 68e679423cba5a5772a0f4bdc61f656e, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-10 13:54:27,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 68e679423cba5a5772a0f4bdc61f656e, disabling compactions & flushes 2023-02-10 13:54:27,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. after waiting 0 ms 2023-02-10 13:54:27,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-10 13:54:27,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:27,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e. 2023-02-10 13:54:27,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 68e679423cba5a5772a0f4bdc61f656e: Waiting for close lock at 1676037267737Running coprocessor pre-close hooks at 1676037267737Disabling compacts and flushes for region at 1676037267737Disabling writes for close at 1676037267738 (+1 ms)Writing region close event to WAL at 1676037267739 (+1 ms)Running coprocessor post-close hooks at 1676037267742 (+3 ms)Closed at 1676037267743 (+1 ms) 2023-02-10 13:54:27,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61e4dbae3501200f091cac3fbd180757, disabling compactions & flushes 2023-02-10 13:54:27,745 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. after waiting 0 ms 2023-02-10 13:54:27,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,745 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=68e679423cba5a5772a0f4bdc61f656e, regionState=CLOSED 2023-02-10 13:54:27,746 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267745"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037267745"}]},"ts":"1676037267745"} 2023-02-10 13:54:27,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-10 13:54:27,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=67, resume processing ppid=64 2023-02-10 13:54:27,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:27,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=64, state=SUCCESS; CloseRegionProcedure 68e679423cba5a5772a0f4bdc61f656e, server=jenkins-hbase4.apache.org,46307,1676037243975 in 158 msec 2023-02-10 13:54:27,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757. 2023-02-10 13:54:27,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61e4dbae3501200f091cac3fbd180757: Waiting for close lock at 1676037267744Running coprocessor pre-close hooks at 1676037267744Disabling compacts and flushes for region at 1676037267744Disabling writes for close at 1676037267745 (+1 ms)Writing region close event to WAL at 1676037267746 (+1 ms)Running coprocessor post-close hooks at 1676037267749 (+3 ms)Closed at 1676037267750 (+1 ms) 2023-02-10 13:54:27,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=68e679423cba5a5772a0f4bdc61f656e, UNASSIGN in 167 msec 2023-02-10 13:54:27,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,757 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=61e4dbae3501200f091cac3fbd180757, regionState=CLOSED 2023-02-10 13:54:27,757 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037267757"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037267757"}]},"ts":"1676037267757"} 2023-02-10 13:54:27,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=66, resume processing ppid=65 2023-02-10 13:54:27,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=65, state=SUCCESS; CloseRegionProcedure 61e4dbae3501200f091cac3fbd180757, server=jenkins-hbase4.apache.org,46307,1676037243975 in 173 msec 2023-02-10 13:54:27,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=63 2023-02-10 13:54:27,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=61e4dbae3501200f091cac3fbd180757, UNASSIGN in 177 msec 2023-02-10 13:54:27,762 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037267762"}]},"ts":"1676037267762"} 2023-02-10 13:54:27,763 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLED in hbase:meta 2023-02-10 13:54:27,765 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionNoStoreFile to state=DISABLED 2023-02-10 13:54:27,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile in 188 msec 2023-02-10 13:54:27,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-10 13:54:27,882 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 63 completed 2023-02-10 13:54:27,883 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,885 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,886 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=68, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-02-10 13:54:27,891 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:27,891 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,891 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,893 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/recovered.edits] 2023-02-10 13:54:27,893 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/recovered.edits] 2023-02-10 13:54:27,893 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/recovered.edits] 2023-02-10 13:54:27,899 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/recovered.edits/4.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6/recovered.edits/4.seqid 2023-02-10 13:54:27,899 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/recovered.edits/7.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e/recovered.edits/7.seqid 2023-02-10 13:54:27,899 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/recovered.edits/7.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757/recovered.edits/7.seqid 2023-02-10 13:54:27,900 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/0cb6d3ddcccf2a26445c39a07e67b5a6 2023-02-10 13:54:27,900 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/68e679423cba5a5772a0f4bdc61f656e 2023-02-10 13:54:27,900 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionNoStoreFile/61e4dbae3501200f091cac3fbd180757 2023-02-10 13:54:27,900 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-02-10 13:54:27,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=68, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,904 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionNoStoreFile from hbase:meta 2023-02-10 13:54:27,906 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionNoStoreFile' descriptor. 2023-02-10 13:54:27,907 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=68, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,907 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionNoStoreFile' from region states. 2023-02-10 13:54:27,907 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037267907"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:27,907 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037267907"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:27,907 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037267907"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:27,908 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:54:27,909 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0cb6d3ddcccf2a26445c39a07e67b5a6, NAME => 'testSplitTableRegionNoStoreFile,,1676037266315.0cb6d3ddcccf2a26445c39a07e67b5a6.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 68e679423cba5a5772a0f4bdc61f656e, NAME => 'testSplitTableRegionNoStoreFile,,1676037266817.68e679423cba5a5772a0f4bdc61f656e.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 61e4dbae3501200f091cac3fbd180757, NAME => 'testSplitTableRegionNoStoreFile,41,1676037266817.61e4dbae3501200f091cac3fbd180757.', STARTKEY => '41', ENDKEY => ''}] 2023-02-10 13:54:27,909 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionNoStoreFile' as deleted. 2023-02-10 13:54:27,909 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037267909"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:27,910 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionNoStoreFile state from META 2023-02-10 13:54:27,913 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=68, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:54:27,914 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile in 30 msec 2023-02-10 13:54:27,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-02-10 13:54:27,987 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 68 completed 2023-02-10 13:54:28,001 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=403 (was 399) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2142237654_15 at /127.0.0.1:37998 [Waiting for operation #41] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2142237654_15 at /127.0.0.1:42100 [Waiting for operation #30] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=866 (was 866), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=227 (was 227), ProcessCount=168 (was 168), AvailableMemoryMB=3435 (was 3438) 2023-02-10 13:54:28,014 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=403, OpenFileDescriptor=866, MaxFileDescriptor=60000, SystemLoadAverage=227, ProcessCount=168, AvailableMemoryMB=3435 2023-02-10 13:54:28,015 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:28,017 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:54:28,017 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:54:28,018 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,020 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:28,020 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:28,022 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,022 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 empty. 2023-02-10 13:54:28,023 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,023 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-02-10 13:54:28,035 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:28,036 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(7671): creating {ENCODED => 61675fc18532b0447c297a024d6bdc99, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionDeletedRowsDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:28,047 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:28,047 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1603): Closing 61675fc18532b0447c297a024d6bdc99, disabling compactions & flushes 2023-02-10 13:54:28,047 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,047 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,047 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. after waiting 0 ms 2023-02-10 13:54:28,047 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,047 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,047 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1557): Region close journal for 61675fc18532b0447c297a024d6bdc99: Waiting for close lock at 1676037268047Disabling compacts and flushes for region at 1676037268047Disabling writes for close at 1676037268047Writing region close event to WAL at 1676037268047Closed at 1676037268047 2023-02-10 13:54:28,049 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:28,050 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1676037268050"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037268050"}]},"ts":"1676037268050"} 2023-02-10 13:54:28,051 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:28,052 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:28,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037268052"}]},"ts":"1676037268052"} 2023-02-10 13:54:28,053 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLING in hbase:meta 2023-02-10 13:54:28,055 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:28,056 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:28,056 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:28,056 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:28,056 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:28,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, ASSIGN}] 2023-02-10 13:54:28,058 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, ASSIGN 2023-02-10 13:54:28,058 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:54:28,209 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:28,210 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=61675fc18532b0447c297a024d6bdc99, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,210 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1676037268210"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037268210"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037268210"}]},"ts":"1676037268210"} 2023-02-10 13:54:28,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=70, state=RUNNABLE; OpenRegionProcedure 61675fc18532b0447c297a024d6bdc99, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:28,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 61675fc18532b0447c297a024d6bdc99, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:28,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:28,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:28,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,371 INFO [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,373 DEBUG [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1 2023-02-10 13:54:28,373 DEBUG [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1 2023-02-10 13:54:28,373 INFO [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61675fc18532b0447c297a024d6bdc99 columnFamilyName cf1 2023-02-10 13:54:28,374 INFO [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] regionserver.HStore(310): Store=61675fc18532b0447c297a024d6bdc99/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:28,374 INFO [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,375 DEBUG [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2 2023-02-10 13:54:28,375 DEBUG [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2 2023-02-10 13:54:28,376 INFO [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61675fc18532b0447c297a024d6bdc99 columnFamilyName cf2 2023-02-10 13:54:28,376 INFO [StoreOpener-61675fc18532b0447c297a024d6bdc99-1] regionserver.HStore(310): Store=61675fc18532b0447c297a024d6bdc99/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:28,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:28,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:28,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:28,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 61675fc18532b0447c297a024d6bdc99; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68500771, jitterRate=0.02074103057384491}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:28,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 61675fc18532b0447c297a024d6bdc99: Running coprocessor pre-open hook at 1676037268370Writing region info on filesystem at 1676037268370Initializing all the Stores at 1676037268371 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037268371Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037268371Cleaning up temporary data from old regions at 1676037268378 (+7 ms)Cleaning up detritus from prior splits at 1676037268379 (+1 ms)Running coprocessor post-open hooks at 1676037268385 (+6 ms)Region opened successfully at 1676037268385 2023-02-10 13:54:28,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99., pid=71, masterSystemTime=1676037268363 2023-02-10 13:54:28,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,388 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=61675fc18532b0447c297a024d6bdc99, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,388 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1676037268388"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037268388"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037268388"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037268388"}]},"ts":"1676037268388"} 2023-02-10 13:54:28,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=70 2023-02-10 13:54:28,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=70, state=SUCCESS; OpenRegionProcedure 61675fc18532b0447c297a024d6bdc99, server=jenkins-hbase4.apache.org,40651,1676037244010 in 179 msec 2023-02-10 13:54:28,395 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=69 2023-02-10 13:54:28,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=69, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, ASSIGN in 337 msec 2023-02-10 13:54:28,397 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:28,397 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037268397"}]},"ts":"1676037268397"} 2023-02-10 13:54:28,398 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLED in hbase:meta 2023-02-10 13:54:28,401 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:28,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 384 msec 2023-02-10 13:54:28,526 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:28,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,528 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:28,528 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,528 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:28,528 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:28,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,530 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,530 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,530 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,531 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,532 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,532 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:28,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,533 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:54:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,553 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/43be20d10284472d90d6e7347e03ab45 2023-02-10 13:54:28,571 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/ddb8723500474913b659135c06d5c549 2023-02-10 13:54:28,577 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/43be20d10284472d90d6e7347e03ab45 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45 2023-02-10 13:54:28,588 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:28,589 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/ddb8723500474913b659135c06d5c549 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549 2023-02-10 13:54:28,593 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:54:28,594 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 61ms, sequenceid=6, compaction requested=false 2023-02-10 13:54:28,594 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:28,595 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:28,595 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,596 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,597 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,597 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f7f36c[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:28,599 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:28,599 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:28,602 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,602 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,604 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:28,608 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:28,608 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:28,608 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:28,608 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,608 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,610 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,612 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,612 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,613 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,614 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,615 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,615 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,621 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,621 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:28,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,621 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,622 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,622 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,722 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:28,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,738 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:28,738 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,738 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:28,738 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,740 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,740 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,740 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:28,742 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,743 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:28,743 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,744 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,745 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,745 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,759 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/a4ef435470f54f66b97f0a4459dd824c 2023-02-10 13:54:28,780 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/b1738299247e4ebfa823e67260488f85 2023-02-10 13:54:28,785 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/a4ef435470f54f66b97f0a4459dd824c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c 2023-02-10 13:54:28,788 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:28,789 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/b1738299247e4ebfa823e67260488f85 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85 2023-02-10 13:54:28,793 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:54:28,794 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 51ms, sequenceid=19, compaction requested=false 2023-02-10 13:54:28,794 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:28,794 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,797 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e64ec91[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:28,800 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:28,800 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:28,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,801 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,802 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:28,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,804 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,804 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,804 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,806 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:28,810 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:28,810 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:28,810 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:28,810 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,810 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:28,811 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,811 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,811 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,811 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,811 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,812 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,813 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,813 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,815 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,815 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,815 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,818 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:28,818 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,818 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,818 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:28,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,819 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,919 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:28,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:28,933 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:28,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,935 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:28,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:28,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:28,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,935 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,936 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,936 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,936 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:28,936 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:28,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:28,938 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:28,938 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:28,938 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,939 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,950 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/45b65037ec8e45a0851d87e4ecdcf932 2023-02-10 13:54:28,962 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/f41de25e213247808f8461da3586170a 2023-02-10 13:54:28,966 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/45b65037ec8e45a0851d87e4ecdcf932 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932 2023-02-10 13:54:28,970 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:28,971 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/f41de25e213247808f8461da3586170a as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a 2023-02-10 13:54:28,975 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:54:28,976 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 37ms, sequenceid=32, compaction requested=true 2023-02-10 13:54:28,976 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:28,976 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:28,976 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:28,981 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77fc4af4[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:28,983 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:28,983 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:28,985 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,985 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:28,987 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,987 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:28,988 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,988 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:28,991 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:28,991 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:28,991 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:28,991 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,991 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,993 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,994 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,994 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,994 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:28,996 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:28,996 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:28,997 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,000 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,000 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:29,000 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,000 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,001 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,101 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:29,115 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:29,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:29,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,116 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,117 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,117 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:29,117 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:29,117 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,117 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,118 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,118 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,118 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,118 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,119 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,119 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,120 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,120 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,120 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:29,121 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:29,121 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,121 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:29,121 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,134 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/9cf83913871e45a4b0c9a978e5ddc959 2023-02-10 13:54:29,146 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/68ce3502c70943f08f4023590e55a6a5 2023-02-10 13:54:29,150 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/9cf83913871e45a4b0c9a978e5ddc959 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959 2023-02-10 13:54:29,154 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:29,154 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/68ce3502c70943f08f4023590e55a6a5 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5 2023-02-10 13:54:29,158 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:54:29,159 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 38ms, sequenceid=45, compaction requested=true 2023-02-10 13:54:29,159 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:29,159 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,161 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,161 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:29,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a7fa54[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:29,164 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:29,164 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,166 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:29,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:29,167 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,167 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:29,167 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,167 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,168 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,169 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,169 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:29,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:29,172 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:29,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:29,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:29,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,172 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:29,177 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,177 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,177 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,177 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,178 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,177 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,178 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,178 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,178 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,179 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,181 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,181 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,184 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,184 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:29,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,184 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,184 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,185 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,185 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,284 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:29,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:29,298 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:29,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:29,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,299 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:29,299 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:29,300 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,300 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,301 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,301 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,301 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,301 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,301 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,301 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:29,303 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,303 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:29,304 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,306 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,306 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,306 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,306 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,306 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,316 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/48196febe2c3448388391680468ad996 2023-02-10 13:54:29,330 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/cbd65cc01d084fb79b88a1ee7513b06c 2023-02-10 13:54:29,336 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/48196febe2c3448388391680468ad996 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996 2023-02-10 13:54:29,341 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:29,342 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/cbd65cc01d084fb79b88a1ee7513b06c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c 2023-02-10 13:54:29,348 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:54:29,349 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 45ms, sequenceid=58, compaction requested=true 2023-02-10 13:54:29,349 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:29,349 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,349 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,351 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,351 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,352 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,352 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:29,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e6189f[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:29,354 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:29,354 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,355 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,355 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,355 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,356 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,356 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:29,356 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,356 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,356 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:29,356 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,359 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,359 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,359 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,359 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,359 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:29,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:29,362 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:29,362 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:29,363 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:29,363 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,363 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:29,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,365 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,364 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,365 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,366 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,367 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,369 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,369 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,369 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,374 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,374 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:29,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,374 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,375 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,374 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,375 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,375 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,375 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,475 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:29,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:29,488 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,489 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,489 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,490 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:29,490 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:29,490 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,490 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,495 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,495 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,495 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,495 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,495 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,495 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:29,497 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:29,497 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,497 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:29,498 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:54:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,506 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/071e19d3c03743e1ab5fd35170251d29 2023-02-10 13:54:29,518 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/4420230fa29f4897b5bf040bb5f3a85b 2023-02-10 13:54:29,522 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/071e19d3c03743e1ab5fd35170251d29 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29 2023-02-10 13:54:29,526 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:29,526 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/4420230fa29f4897b5bf040bb5f3a85b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b 2023-02-10 13:54:29,530 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:54:29,531 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 33ms, sequenceid=71, compaction requested=true 2023-02-10 13:54:29,531 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:29,531 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,531 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:29,531 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:29,531 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,534 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,534 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,534 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,534 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:29,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ddf64f6[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:29,536 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:29,536 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,537 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,538 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,537 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,538 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:29,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,537 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,538 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,538 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,537 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,538 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:29,538 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:29,538 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,539 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,539 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,540 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:29,543 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:29,543 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:29,543 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:29,543 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,543 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:29,545 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,546 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,547 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,547 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,548 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,549 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,549 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,551 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,551 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,551 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,551 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,552 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,652 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:29,666 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:29,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:29,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,667 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,667 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,668 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:29,668 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:29,668 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,668 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,669 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,669 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,669 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,669 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,670 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,670 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,671 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,672 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,672 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,672 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:29,672 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:29,672 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,672 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,672 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,672 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:29,672 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=530 B heapSize=2.33 KB 2023-02-10 13:54:29,673 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,675 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,675 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,675 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,675 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,675 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,681 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:29,685 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:29,693 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:29,696 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:29,697 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/355e2783c02648359e715942fdd8fb7e as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:29,701 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:29,701 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e, entries=9, sequenceid=83, filesize=5.1 K 2023-02-10 13:54:29,702 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:29,705 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:29,706 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8, entries=9, sequenceid=83, filesize=5.1 K 2023-02-10 13:54:29,706 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~530 B/530, heapSize ~2.30 KB/2352, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 34ms, sequenceid=83, compaction requested=true 2023-02-10 13:54:29,707 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:29,707 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,707 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,708 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,709 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68613a53[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:29,710 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:29,710 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,711 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,711 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,711 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,712 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,712 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:29,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:29,712 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:29,716 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,716 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,716 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,716 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,716 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,717 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:29,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:29,720 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:29,720 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:29,720 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:29,720 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,720 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,723 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,722 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,723 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,723 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,725 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,725 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,725 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,728 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,728 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,728 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:29,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,729 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,829 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:29,838 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,839 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,839 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,839 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:29,839 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:29,839 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,840 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,842 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,842 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,842 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,842 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,843 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,844 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:29,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:29,845 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,845 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:29,845 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-02-10 13:54:29,845 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,846 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,854 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:29,857 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:29,864 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:29,867 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:29,868 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/bd08426725c14683a929b013b2822c9e as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:29,872 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:29,872 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e, entries=5, sequenceid=91, filesize=5.0 K 2023-02-10 13:54:29,872 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/c501090c1e8e4d0e88b16e16c320166a as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:29,876 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:29,876 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a, entries=5, sequenceid=91, filesize=5.0 K 2023-02-10 13:54:29,877 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 32ms, sequenceid=91, compaction requested=true 2023-02-10 13:54:29,877 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:29,877 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:29,879 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:29,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec20851[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:29,881 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:29,881 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,883 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,883 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:29,883 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,885 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,885 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,886 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,886 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:29,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:29,888 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:29,888 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:29,888 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:29,888 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,889 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,890 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,890 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,891 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:29,891 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:29,893 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,893 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,893 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,900 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:29,900 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,900 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:29,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,900 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:29,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,001 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:30,009 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:54:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:54:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,011 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:30,011 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:30,011 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-10 13:54:30,011 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-10 13:54:30,011 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,011 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,012 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:30,012 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,012 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:30,012 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:30,013 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,013 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:54:30,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:30,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:30,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:30,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:30,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:30,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-10 13:54:30,014 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:30,014 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. started... 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:30,014 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:30,023 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:30,027 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:30,034 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/212676eabea14142a24806574fb89722 2023-02-10 13:54:30,037 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 212676eabea14142a24806574fb89722 2023-02-10 13:54:30,038 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/4a536c11616e4cf49789c92f9bae862c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:30,042 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:30,042 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c, entries=5, sequenceid=99, filesize=5.0 K 2023-02-10 13:54:30,043 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/212676eabea14142a24806574fb89722 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722 2023-02-10 13:54:30,046 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 212676eabea14142a24806574fb89722 2023-02-10 13:54:30,046 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722, entries=5, sequenceid=99, filesize=5.0 K 2023-02-10 13:54:30,047 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 33ms, sequenceid=99, compaction requested=true 2023-02-10 13:54:30,047 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 61675fc18532b0447c297a024d6bdc99: 2023-02-10 13:54:30,047 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-10 13:54:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-10 13:54:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,049 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:54:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:30,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-10 13:54:30,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a14cdf6[Count = 0] remaining members to acquire global barrier 2023-02-10 13:54:30,051 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-10 13:54:30,051 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,052 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,052 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,052 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:30,053 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,053 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:54:30,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:54:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-10 13:54:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:54:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:30,055 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,055 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-10 13:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:30,055 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:30,055 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-10 13:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:54:30,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:54:30,057 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-10 13:54:30,058 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:54:30,058 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:54:30,058 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,058 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:54:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:30,059 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,059 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:54:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:30,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:30,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,061 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:54:30,062 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,062 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:30,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:30,068 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:54:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:30,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:30,068 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-10 13:54:30,068 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:54:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:54:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:54:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:30,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:54:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:54:30,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:54:30,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:30,169 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:54:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:54:30,171 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,172 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:30,174 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:30,174 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:30,174 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:30,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, UNASSIGN}] 2023-02-10 13:54:30,176 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, UNASSIGN 2023-02-10 13:54:30,176 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=61675fc18532b0447c297a024d6bdc99, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:30,177 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1676037270176"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037270176"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037270176"}]},"ts":"1676037270176"} 2023-02-10 13:54:30,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=73, state=RUNNABLE; CloseRegionProcedure 61675fc18532b0447c297a024d6bdc99, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:30,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 61675fc18532b0447c297a024d6bdc99, disabling compactions & flushes 2023-02-10 13:54:30,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:30,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:30,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. after waiting 0 ms 2023-02-10 13:54:30,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:30,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 61675fc18532b0447c297a024d6bdc99 2/2 column families, dataSize=50 B heapSize=720 B 2023-02-10 13:54:30,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:30,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:30,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:30,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:30,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf1/3f6e762c937e468ba0d22b2177d9e958 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:30,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:30,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958, entries=1, sequenceid=103, filesize=4.9 K 2023-02-10 13:54:30,362 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.tmp/cf2/8eb040f3ce1e4cd39ccd419705083c98 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:30,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:30,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98, entries=1, sequenceid=103, filesize=4.9 K 2023-02-10 13:54:30,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 36ms, sequenceid=103, compaction requested=true 2023-02-10 13:54:30,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=1 2023-02-10 13:54:30,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:30,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. 2023-02-10 13:54:30,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 61675fc18532b0447c297a024d6bdc99: Waiting for close lock at 1676037270330Running coprocessor pre-close hooks at 1676037270330Disabling compacts and flushes for region at 1676037270330Disabling writes for close at 1676037270330Obtaining lock to block concurrent updates at 1676037270330Preparing flush snapshotting stores in 61675fc18532b0447c297a024d6bdc99 at 1676037270330Finished memstore snapshotting testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99., syncing WAL and waiting on mvcc, flushsize=dataSize=50, getHeapSize=688, getOffHeapSize=0, getCellsCount=2 at 1676037270330Flushing stores of testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99. at 1676037270331 (+1 ms)Flushing 61675fc18532b0447c297a024d6bdc99/cf1: creating writer at 1676037270332 (+1 ms)Flushing 61675fc18532b0447c297a024d6bdc99/cf1: appending metadata at 1676037270334 (+2 ms)Flushing 61675fc18532b0447c297a024d6bdc99/cf1: closing flushed file at 1676037270334Flushing 61675fc18532b0447c297a024d6bdc99/cf2: creating writer at 1676037270345 (+11 ms)Flushing 61675fc18532b0447c297a024d6bdc99/cf2: appending metadata at 1676037270347 (+2 ms)Flushing 61675fc18532b0447c297a024d6bdc99/cf2: closing flushed file at 1676037270347Flushing 61675fc18532b0447c297a024d6bdc99/cf1: reopening flushed file at 1676037270358 (+11 ms)Flushing 61675fc18532b0447c297a024d6bdc99/cf2: reopening flushed file at 1676037270362 (+4 ms)Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for 61675fc18532b0447c297a024d6bdc99 in 36ms, sequenceid=103, compaction requested=true at 1676037270366 (+4 ms)Writing region close event to WAL at 1676037270369 (+3 ms)Running coprocessor post-close hooks at 1676037270372 (+3 ms)Closed at 1676037270372 2023-02-10 13:54:30,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,374 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=61675fc18532b0447c297a024d6bdc99, regionState=CLOSED 2023-02-10 13:54:30,374 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1676037270374"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037270374"}]},"ts":"1676037270374"} 2023-02-10 13:54:30,376 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=73 2023-02-10 13:54:30,376 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=73, state=SUCCESS; CloseRegionProcedure 61675fc18532b0447c297a024d6bdc99, server=jenkins-hbase4.apache.org,40651,1676037244010 in 197 msec 2023-02-10 13:54:30,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=72 2023-02-10 13:54:30,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=61675fc18532b0447c297a024d6bdc99, UNASSIGN in 202 msec 2023-02-10 13:54:30,391 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=72 splitting 20 storefiles, region=61675fc18532b0447c297a024d6bdc99, threads=16 2023-02-10 13:54:30,391 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,391 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,391 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,391 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,392 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,398 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,399 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,399 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,399 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,399 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,402 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,402 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,402 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,403 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,403 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,406 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,409 INFO [StoreFileSplitter-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 212676eabea14142a24806574fb89722 2023-02-10 13:54:30,411 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c, top=false 2023-02-10 13:54:30,413 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5, top=false 2023-02-10 13:54:30,415 INFO [StoreFileSplitter-pool-8] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:30,418 INFO [StoreFileSplitter-pool-3] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:30,419 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b, top=false 2023-02-10 13:54:30,419 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:30,420 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29, top=false 2023-02-10 13:54:30,421 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:30,424 DEBUG [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996, top=false 2023-02-10 13:54:30,425 INFO [StoreFileSplitter-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 212676eabea14142a24806574fb89722 2023-02-10 13:54:30,425 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722, top=true 2023-02-10 13:54:30,425 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:30,425 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a, top=true 2023-02-10 13:54:30,433 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:30,443 INFO [StoreFileSplitter-pool-3] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:30,443 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98, top=true 2023-02-10 13:54:30,446 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5 for child: fa1d3c7bf2e8456cfebc77cba079ea0c, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,449 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a for child: b6601c94d8fe2c319f145aba750fb95d, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,449 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,449 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29 for child: fa1d3c7bf2e8456cfebc77cba079ea0c, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,449 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,451 INFO [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996 for child: fa1d3c7bf2e8456cfebc77cba079ea0c, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,451 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98 for child: b6601c94d8fe2c319f145aba750fb95d, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,451 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,452 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,453 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c for child: fa1d3c7bf2e8456cfebc77cba079ea0c, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,453 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b for child: fa1d3c7bf2e8456cfebc77cba079ea0c, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,456 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:30,456 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,456 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,457 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,457 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,460 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,462 INFO [StoreFileSplitter-pool-15] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:30,463 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:30,463 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c, top=true 2023-02-10 13:54:30,463 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,465 INFO [StoreFileSplitter-pool-15] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:30,465 DEBUG [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e, top=true 2023-02-10 13:54:30,472 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722 for child: b6601c94d8fe2c319f145aba750fb95d, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,473 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,479 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:30,479 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958, top=true 2023-02-10 13:54:30,490 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,491 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959, top=false 2023-02-10 13:54:30,499 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c for child: b6601c94d8fe2c319f145aba750fb95d, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,499 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,503 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:30,508 INFO [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e for child: b6601c94d8fe2c319f145aba750fb95d, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,508 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,509 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958 for child: b6601c94d8fe2c319f145aba750fb95d, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,509 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,515 INFO [StoreFileSplitter-pool-8] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:30,516 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/.splits/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959 for child: fa1d3c7bf2e8456cfebc77cba079ea0c, parent: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,521 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,531 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,531 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,532 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,543 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,543 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,916 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549 for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,919 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,920 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e for region: 61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:30,920 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=72 split storefiles for region 61675fc18532b0447c297a024d6bdc99 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:54:30,958 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-02-10 13:54:30,960 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-02-10 13:54:30,961 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1676037270961"},{"qualifier":"splitA","vlen":75,"tag":[],"timestamp":"1676037270961"},{"qualifier":"splitB","vlen":75,"tag":[],"timestamp":"1676037270961"}]},"ts":"1676037270961"} 2023-02-10 13:54:30,961 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037270961"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037270961"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037270961"}]},"ts":"1676037270961"} 2023-02-10 13:54:30,961 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037270961"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037270961"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037270961"}]},"ts":"1676037270961"} 2023-02-10 13:54:30,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, ASSIGN}, {pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, ASSIGN}] 2023-02-10 13:54:30,965 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, ASSIGN 2023-02-10 13:54:30,966 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:54:30,966 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, ASSIGN 2023-02-10 13:54:30,967 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:54:31,116 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:54:31,118 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=fa1d3c7bf2e8456cfebc77cba079ea0c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:31,118 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037271118"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037271118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037271118"}]},"ts":"1676037271118"} 2023-02-10 13:54:31,120 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=75, state=RUNNABLE; OpenRegionProcedure fa1d3c7bf2e8456cfebc77cba079ea0c, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:31,122 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=b6601c94d8fe2c319f145aba750fb95d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:31,122 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037271122"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037271122"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037271122"}]},"ts":"1676037271122"} 2023-02-10 13:54:31,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=78, ppid=76, state=RUNNABLE; OpenRegionProcedure b6601c94d8fe2c319f145aba750fb95d, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:31,231 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:54:31,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fa1d3c7bf2e8456cfebc77cba079ea0c, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.', STARTKEY => '', ENDKEY => '60'} 2023-02-10 13:54:31,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:31,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:31,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,277 INFO [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,277 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 2023-02-10 13:54:31,277 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 2023-02-10 13:54:31,278 INFO [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1d3c7bf2e8456cfebc77cba079ea0c columnFamilyName cf1 2023-02-10 13:54:31,289 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,290 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e-bottom 2023-02-10 13:54:31,296 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45-bottom 2023-02-10 13:54:31,301 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932-bottom 2023-02-10 13:54:31,305 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c-bottom 2023-02-10 13:54:31,310 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29 2023-02-10 13:54:31,313 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996 2023-02-10 13:54:31,317 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959 2023-02-10 13:54:31,318 INFO [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(310): Store=fa1d3c7bf2e8456cfebc77cba079ea0c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:31,318 INFO [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,319 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 2023-02-10 13:54:31,319 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 2023-02-10 13:54:31,319 INFO [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1d3c7bf2e8456cfebc77cba079ea0c columnFamilyName cf2 2023-02-10 13:54:31,331 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85-bottom 2023-02-10 13:54:31,336 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549-bottom 2023-02-10 13:54:31,340 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,341 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8-bottom 2023-02-10 13:54:31,346 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a-bottom 2023-02-10 13:54:31,349 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b 2023-02-10 13:54:31,352 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5 2023-02-10 13:54:31,355 DEBUG [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c 2023-02-10 13:54:31,355 INFO [StoreOpener-fa1d3c7bf2e8456cfebc77cba079ea0c-1] regionserver.HStore(310): Store=fa1d3c7bf2e8456cfebc77cba079ea0c/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:31,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:31,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened fa1d3c7bf2e8456cfebc77cba079ea0c; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61216051, jitterRate=-0.08780975639820099}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:31,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for fa1d3c7bf2e8456cfebc77cba079ea0c: Running coprocessor pre-open hook at 1676037271276Writing region info on filesystem at 1676037271276Initializing all the Stores at 1676037271276Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037271276Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037271276Cleaning up temporary data from old regions at 1676037271358 (+82 ms)Cleaning up detritus from prior splits at 1676037271358Running coprocessor post-open hooks at 1676037271361 (+3 ms)Region opened successfully at 1676037271361 2023-02-10 13:54:31,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c., pid=77, masterSystemTime=1676037271271 2023-02-10 13:54:31,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:31,362 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:31,363 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:31,363 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,363 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,363 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:31,363 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:31,363 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 in testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,363 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 in testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,363 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp, totalSize=35.0 K 2023-02-10 13:54:31,363 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp, totalSize=35.0 K 2023-02-10 13:54:31,364 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037268521 2023-02-10 13:54:31,364 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037268521 2023-02-10 13:54:31,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,364 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037268724 2023-02-10 13:54:31,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,364 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037268724 2023-02-10 13:54:31,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6601c94d8fe2c319f145aba750fb95d, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.', STARTKEY => '60', ENDKEY => ''} 2023-02-10 13:54:31,365 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=fa1d3c7bf2e8456cfebc77cba079ea0c, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:31,365 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037271365"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037271365"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037271365"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037271365"}]},"ts":"1676037271365"} 2023-02-10 13:54:31,365 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037268921 2023-02-10 13:54:31,365 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037268921 2023-02-10 13:54:31,365 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037269103 2023-02-10 13:54:31,365 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037269103 2023-02-10 13:54:31,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:31,366 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037269286 2023-02-10 13:54:31,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:31,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,366 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037269286 2023-02-10 13:54:31,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,366 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037269477 2023-02-10 13:54:31,366 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037269477 2023-02-10 13:54:31,367 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037269654 2023-02-10 13:54:31,367 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037269654 2023-02-10 13:54:31,368 INFO [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=77, resume processing ppid=75 2023-02-10 13:54:31,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=75, state=SUCCESS; OpenRegionProcedure fa1d3c7bf2e8456cfebc77cba079ea0c, server=jenkins-hbase4.apache.org,40651,1676037244010 in 246 msec 2023-02-10 13:54:31,370 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1 2023-02-10 13:54:31,370 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1 2023-02-10 13:54:31,370 INFO [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6601c94d8fe2c319f145aba750fb95d columnFamilyName cf1 2023-02-10 13:54:31,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, ASSIGN in 405 msec 2023-02-10 13:54:31,383 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,383 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e-top 2023-02-10 13:54:31,384 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1d3c7bf2e8456cfebc77cba079ea0c#cf2#compaction#72 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,384 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa1d3c7bf2e8456cfebc77cba079ea0c#cf1#compaction#73 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,388 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45-top 2023-02-10 13:54:31,392 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932-top 2023-02-10 13:54:31,402 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c-top 2023-02-10 13:54:31,408 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:31,408 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:31,409 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp/cf2/defba3f78eae4033b96ad56ee6a6a6ce as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/defba3f78eae4033b96ad56ee6a6a6ce 2023-02-10 13:54:31,411 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp/cf1/fd507a6eb3bc4e9b8b180518622624a4 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/fd507a6eb3bc4e9b8b180518622624a4 2023-02-10 13:54:31,412 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:31,412 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:31,415 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 of fa1d3c7bf2e8456cfebc77cba079ea0c into defba3f78eae4033b96ad56ee6a6a6ce(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,415 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for fa1d3c7bf2e8456cfebc77cba079ea0c: 2023-02-10 13:54:31,415 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c., storeName=fa1d3c7bf2e8456cfebc77cba079ea0c/cf2, priority=9, startTime=1676037271362; duration=0sec 2023-02-10 13:54:31,415 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,416 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 of fa1d3c7bf2e8456cfebc77cba079ea0c into fd507a6eb3bc4e9b8b180518622624a4(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,416 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for fa1d3c7bf2e8456cfebc77cba079ea0c: 2023-02-10 13:54:31,416 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:31,416 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c., storeName=fa1d3c7bf2e8456cfebc77cba079ea0c/cf1, priority=9, startTime=1676037271361; duration=0sec 2023-02-10 13:54:31,416 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:31,416 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,416 INFO [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(310): Store=b6601c94d8fe2c319f145aba750fb95d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:31,416 INFO [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,417 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2 2023-02-10 13:54:31,417 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2 2023-02-10 13:54:31,417 INFO [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6601c94d8fe2c319f145aba750fb95d columnFamilyName cf2 2023-02-10 13:54:31,426 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85-top 2023-02-10 13:54:31,428 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549-top 2023-02-10 13:54:31,431 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,431 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8-top 2023-02-10 13:54:31,434 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a-top 2023-02-10 13:54:31,437 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 212676eabea14142a24806574fb89722 2023-02-10 13:54:31,437 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722 2023-02-10 13:54:31,440 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:31,440 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:31,443 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:31,444 DEBUG [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:31,444 INFO [StoreOpener-b6601c94d8fe2c319f145aba750fb95d-1] regionserver.HStore(310): Store=b6601c94d8fe2c319f145aba750fb95d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:31,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:54:31,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:31,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened b6601c94d8fe2c319f145aba750fb95d; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61833815, jitterRate=-0.07860435545444489}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:54:31,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for b6601c94d8fe2c319f145aba750fb95d: Running coprocessor pre-open hook at 1676037271366Writing region info on filesystem at 1676037271366Initializing all the Stores at 1676037271368 (+2 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037271368Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037271368Cleaning up temporary data from old regions at 1676037271446 (+78 ms)Cleaning up detritus from prior splits at 1676037271446Running coprocessor post-open hooks at 1676037271449 (+3 ms)Region opened successfully at 1676037271449 2023-02-10 13:54:31,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d., pid=78, masterSystemTime=1676037271271 2023-02-10 13:54:31,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:31,452 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:31,452 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:54:31,453 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,453 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,453 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): b6601c94d8fe2c319f145aba750fb95d/cf1 is initiating minor compaction (all files) 2023-02-10 13:54:31,453 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): b6601c94d8fe2c319f145aba750fb95d/cf2 is initiating minor compaction (all files) 2023-02-10 13:54:31,453 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of b6601c94d8fe2c319f145aba750fb95d/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,453 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of b6601c94d8fe2c319f145aba750fb95d/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,453 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp, totalSize=34.8 K 2023-02-10 13:54:31,453 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp, totalSize=34.8 K 2023-02-10 13:54:31,453 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037268521 2023-02-10 13:54:31,454 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037268521 2023-02-10 13:54:31,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,454 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037268724 2023-02-10 13:54:31,454 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037268724 2023-02-10 13:54:31,454 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=b6601c94d8fe2c319f145aba750fb95d, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:31,454 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037268921 2023-02-10 13:54:31,454 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037271454"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037271454"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037271454"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037271454"}]},"ts":"1676037271454"} 2023-02-10 13:54:31,455 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037268921 2023-02-10 13:54:31,455 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1676037269654 2023-02-10 13:54:31,455 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1676037269654 2023-02-10 13:54:31,455 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,455 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,456 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,456 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,456 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,456 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=78, resume processing ppid=76 2023-02-10 13:54:31,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; OpenRegionProcedure b6601c94d8fe2c319f145aba750fb95d, server=jenkins-hbase4.apache.org,40651,1676037244010 in 333 msec 2023-02-10 13:54:31,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=76, resume processing ppid=72 2023-02-10 13:54:31,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, ASSIGN in 494 msec 2023-02-10 13:54:31,461 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d in 1.2880 sec 2023-02-10 13:54:31,474 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): b6601c94d8fe2c319f145aba750fb95d#cf2#compaction#75 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,474 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6601c94d8fe2c319f145aba750fb95d#cf1#compaction#74 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,494 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 1ebcf0effd7041a5992f79994a074188: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,495 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp/cf2/1ebcf0effd7041a5992f79994a074188 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/1ebcf0effd7041a5992f79994a074188 2023-02-10 13:54:31,495 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for ea71ff687b4a4187a42bcaa0725a2fda: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,496 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp/cf1/ea71ff687b4a4187a42bcaa0725a2fda as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/ea71ff687b4a4187a42bcaa0725a2fda 2023-02-10 13:54:31,499 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 1ebcf0effd7041a5992f79994a074188: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,500 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for ea71ff687b4a4187a42bcaa0725a2fda: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,500 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b6601c94d8fe2c319f145aba750fb95d/cf2 of b6601c94d8fe2c319f145aba750fb95d into 1ebcf0effd7041a5992f79994a074188(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,500 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for b6601c94d8fe2c319f145aba750fb95d: 2023-02-10 13:54:31,500 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d., storeName=b6601c94d8fe2c319f145aba750fb95d/cf2, priority=9, startTime=1676037271452; duration=0sec 2023-02-10 13:54:31,500 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,500 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b6601c94d8fe2c319f145aba750fb95d/cf1 of b6601c94d8fe2c319f145aba750fb95d into ea71ff687b4a4187a42bcaa0725a2fda(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,501 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for b6601c94d8fe2c319f145aba750fb95d: 2023-02-10 13:54:31,501 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d., storeName=b6601c94d8fe2c319f145aba750fb95d/cf1, priority=9, startTime=1676037271452; duration=0sec 2023-02-10 13:54:31,501 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,683 DEBUG [Listener at localhost/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-10 13:54:31,687 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43460, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-10 13:54:31,688 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,689 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-10 13:54:31,689 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HStore(1912): fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 is initiating major compaction (all files) 2023-02-10 13:54:31,689 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@2459b428; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,689 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 in testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,689 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/fd507a6eb3bc4e9b8b180518622624a4] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp, totalSize=6.8 K 2023-02-10 13:54:31,689 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-10 13:54:31,690 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HStore(1912): fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 is initiating major compaction (all files) 2023-02-10 13:54:31,690 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@42a2e739; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:31,690 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 in testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,690 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/defba3f78eae4033b96ad56ee6a6a6ce] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp, totalSize=6.8 K 2023-02-10 13:54:31,690 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting fd507a6eb3bc4e9b8b180518622624a4, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037268521 2023-02-10 13:54:31,690 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting defba3f78eae4033b96ad56ee6a6a6ce, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037268521 2023-02-10 13:54:31,692 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,692 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-10 13:54:31,692 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.HStore(1912): b6601c94d8fe2c319f145aba750fb95d/cf1 is initiating major compaction (all files) 2023-02-10 13:54:31,692 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@2ef66eab; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:31,692 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-10 13:54:31,692 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.HStore(1912): b6601c94d8fe2c319f145aba750fb95d/cf2 is initiating major compaction (all files) 2023-02-10 13:54:31,692 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@5e737b62; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-02-10 13:54:31,693 INFO [Listener at localhost/43477] hbase.Waiter(180): Waiting up to [6,000] milli-secs(wait.for.ratio=[1]) 2023-02-10 13:54:31,694 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1d3c7bf2e8456cfebc77cba079ea0c#cf2#compaction#76 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,694 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa1d3c7bf2e8456cfebc77cba079ea0c#cf1#compaction#77 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,706 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. is not mergeable because it has references 2023-02-10 13:54:31,706 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. is not mergeable because it has references 2023-02-10 13:54:31,708 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp/cf2/e48f27de64a64c3b99fbcb2218e0f33c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/e48f27de64a64c3b99fbcb2218e0f33c 2023-02-10 13:54:31,708 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/.tmp/cf1/a13923017bea4c5386a896c8534246c1 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a13923017bea4c5386a896c8534246c1 2023-02-10 13:54:31,713 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in fa1d3c7bf2e8456cfebc77cba079ea0c/cf2 of fa1d3c7bf2e8456cfebc77cba079ea0c into e48f27de64a64c3b99fbcb2218e0f33c(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,713 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for fa1d3c7bf2e8456cfebc77cba079ea0c: 2023-02-10 13:54:31,713 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in fa1d3c7bf2e8456cfebc77cba079ea0c/cf1 of fa1d3c7bf2e8456cfebc77cba079ea0c into a13923017bea4c5386a896c8534246c1(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,713 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for fa1d3c7bf2e8456cfebc77cba079ea0c: 2023-02-10 13:54:31,714 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c., storeName=cf2, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1676037271689; duration=0sec 2023-02-10 13:54:31,714 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c., storeName=cf1, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1676037271689; duration=0sec 2023-02-10 13:54:31,715 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-02-10 13:54:31,715 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:54:31,715 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of b6601c94d8fe2c319f145aba750fb95d/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,715 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of b6601c94d8fe2c319f145aba750fb95d/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:31,715 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/1ebcf0effd7041a5992f79994a074188] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp, totalSize=5.0 K 2023-02-10 13:54:31,715 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/ea71ff687b4a4187a42bcaa0725a2fda] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp, totalSize=5.0 K 2023-02-10 13:54:31,715 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 1ebcf0effd7041a5992f79994a074188, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,715 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting ea71ff687b4a4187a42bcaa0725a2fda, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-10 13:54:31,723 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): b6601c94d8fe2c319f145aba750fb95d#cf2#compaction#78 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,723 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6601c94d8fe2c319f145aba750fb95d#cf1#compaction#79 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:54:31,739 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for c22d6793c3194a6b895f7c13b53cc37f: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,739 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for f84459a5536443068795ed09927e5489: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,740 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp/cf1/c22d6793c3194a6b895f7c13b53cc37f as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/c22d6793c3194a6b895f7c13b53cc37f 2023-02-10 13:54:31,740 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/.tmp/cf2/f84459a5536443068795ed09927e5489 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f84459a5536443068795ed09927e5489 2023-02-10 13:54:31,744 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for c22d6793c3194a6b895f7c13b53cc37f: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,746 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for f84459a5536443068795ed09927e5489: NONE, but ROW specified in column family configuration 2023-02-10 13:54:31,746 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in b6601c94d8fe2c319f145aba750fb95d/cf1 of b6601c94d8fe2c319f145aba750fb95d into c22d6793c3194a6b895f7c13b53cc37f(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,746 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for b6601c94d8fe2c319f145aba750fb95d: 2023-02-10 13:54:31,746 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d., storeName=cf1, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1676037271692; duration=0sec 2023-02-10 13:54:31,746 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,747 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in b6601c94d8fe2c319f145aba750fb95d/cf2 of b6601c94d8fe2c319f145aba750fb95d into f84459a5536443068795ed09927e5489(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:54:31,747 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for b6601c94d8fe2c319f145aba750fb95d: 2023-02-10 13:54:31,747 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d., storeName=cf2, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1676037271692; duration=0sec 2023-02-10 13:54:31,747 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:54:31,814 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. is not mergeable because it has references 2023-02-10 13:54:31,815 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40651] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. is not mergeable because it has references 2023-02-10 13:54:31,821 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '60', inclusive 2023-02-10 13:54:31,823 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:31,824 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:31,824 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:31,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=79, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-02-10 13:54:31,828 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037271828"}]},"ts":"1676037271828"} 2023-02-10 13:54:31,829 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLING in hbase:meta 2023-02-10 13:54:31,830 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLING 2023-02-10 13:54:31,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, UNASSIGN}, {pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, UNASSIGN}] 2023-02-10 13:54:31,831 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, UNASSIGN 2023-02-10 13:54:31,832 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=b6601c94d8fe2c319f145aba750fb95d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:31,832 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037271832"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037271832"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037271832"}]},"ts":"1676037271832"} 2023-02-10 13:54:31,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=82, ppid=81, state=RUNNABLE; CloseRegionProcedure b6601c94d8fe2c319f145aba750fb95d, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:31,835 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, UNASSIGN 2023-02-10 13:54:31,836 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=fa1d3c7bf2e8456cfebc77cba079ea0c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:31,836 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037271836"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037271836"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037271836"}]},"ts":"1676037271836"} 2023-02-10 13:54:31,837 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure fa1d3c7bf2e8456cfebc77cba079ea0c, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:31,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-02-10 13:54:31,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:31,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing fa1d3c7bf2e8456cfebc77cba079ea0c, disabling compactions & flushes 2023-02-10 13:54:31,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. after waiting 0 ms 2023-02-10 13:54:31,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:31,986 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/fd507a6eb3bc4e9b8b180518622624a4, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e-bottom] to archive 2023-02-10 13:54:31,987 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:31,988 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,989 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,990 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,991 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-9cf83913871e45a4b0c9a978e5ddc959 2023-02-10 13:54:31,992 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-48196febe2c3448388391680468ad996 2023-02-10 13:54:31,993 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-071e19d3c03743e1ab5fd35170251d29 2023-02-10 13:54:31,994 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/fd507a6eb3bc4e9b8b180518622624a4 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/fd507a6eb3bc4e9b8b180518622624a4 2023-02-10 13:54:31,995 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:31,997 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/defba3f78eae4033b96ad56ee6a6a6ce, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8-bottom] to archive 2023-02-10 13:54:31,997 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:31,999 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,000 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,001 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,002 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-68ce3502c70943f08f4023590e55a6a5 2023-02-10 13:54:32,003 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-cbd65cc01d084fb79b88a1ee7513b06c 2023-02-10 13:54:32,004 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4420230fa29f4897b5bf040bb5f3a85b 2023-02-10 13:54:32,005 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/defba3f78eae4033b96ad56ee6a6a6ce to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/defba3f78eae4033b96ad56ee6a6a6ce 2023-02-10 13:54:32,006 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-02-10 13:54:32,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:32,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c. 2023-02-10 13:54:32,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for fa1d3c7bf2e8456cfebc77cba079ea0c: Waiting for close lock at 1676037271985Running coprocessor pre-close hooks at 1676037271985Disabling compacts and flushes for region at 1676037271985Disabling writes for close at 1676037271985Writing region close event to WAL at 1676037272007 (+22 ms)Running coprocessor post-close hooks at 1676037272010 (+3 ms)Closed at 1676037272010 2023-02-10 13:54:32,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:32,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:32,011 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing b6601c94d8fe2c319f145aba750fb95d, disabling compactions & flushes 2023-02-10 13:54:32,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:32,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:32,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. after waiting 0 ms 2023-02-10 13:54:32,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:32,012 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=fa1d3c7bf2e8456cfebc77cba079ea0c, regionState=CLOSED 2023-02-10 13:54:32,012 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037272012"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037272012"}]},"ts":"1676037272012"} 2023-02-10 13:54:32,013 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/ea71ff687b4a4187a42bcaa0725a2fda, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958] to archive 2023-02-10 13:54:32,014 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:32,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=80 2023-02-10 13:54:32,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure fa1d3c7bf2e8456cfebc77cba079ea0c, server=jenkins-hbase4.apache.org,40651,1676037244010 in 176 msec 2023-02-10 13:54:32,015 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/43be20d10284472d90d6e7347e03ab45.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=fa1d3c7bf2e8456cfebc77cba079ea0c, UNASSIGN in 184 msec 2023-02-10 13:54:32,017 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/a4ef435470f54f66b97f0a4459dd824c.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,018 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/45b65037ec8e45a0851d87e4ecdcf932.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,019 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/355e2783c02648359e715942fdd8fb7e.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,020 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:32,021 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:32,021 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/ea71ff687b4a4187a42bcaa0725a2fda to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/ea71ff687b4a4187a42bcaa0725a2fda 2023-02-10 13:54:32,023 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:32,025 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/1ebcf0effd7041a5992f79994a074188, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98] to archive 2023-02-10 13:54:32,025 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:54:32,026 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ddb8723500474913b659135c06d5c549.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,027 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/b1738299247e4ebfa823e67260488f85.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,028 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f41de25e213247808f8461da3586170a.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,029 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8.61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,030 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:32,030 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-212676eabea14142a24806574fb89722 2023-02-10 13:54:32,031 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/1ebcf0effd7041a5992f79994a074188 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/1ebcf0effd7041a5992f79994a074188 2023-02-10 13:54:32,032 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/testSplitTableRegionDeletedRowsDaughter=61675fc18532b0447c297a024d6bdc99-8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:32,035 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-02-10 13:54:32,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:32,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d. 2023-02-10 13:54:32,036 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for b6601c94d8fe2c319f145aba750fb95d: Waiting for close lock at 1676037272011Running coprocessor pre-close hooks at 1676037272011Disabling compacts and flushes for region at 1676037272011Disabling writes for close at 1676037272012 (+1 ms)Writing region close event to WAL at 1676037272033 (+21 ms)Running coprocessor post-close hooks at 1676037272036 (+3 ms)Closed at 1676037272036 2023-02-10 13:54:32,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:32,037 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=b6601c94d8fe2c319f145aba750fb95d, regionState=CLOSED 2023-02-10 13:54:32,038 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1676037272037"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037272037"}]},"ts":"1676037272037"} 2023-02-10 13:54:32,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=81 2023-02-10 13:54:32,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=81, state=SUCCESS; CloseRegionProcedure b6601c94d8fe2c319f145aba750fb95d, server=jenkins-hbase4.apache.org,40651,1676037244010 in 206 msec 2023-02-10 13:54:32,041 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=79 2023-02-10 13:54:32,042 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=b6601c94d8fe2c319f145aba750fb95d, UNASSIGN in 209 msec 2023-02-10 13:54:32,042 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037272042"}]},"ts":"1676037272042"} 2023-02-10 13:54:32,043 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLED in hbase:meta 2023-02-10 13:54:32,045 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLED 2023-02-10 13:54:32,047 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 222 msec 2023-02-10 13:54:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-02-10 13:54:32,130 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 79 completed 2023-02-10 13:54:32,130 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,133 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,133 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=84, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-02-10 13:54:32,137 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,137 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:32,137 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:32,139 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/recovered.edits] 2023-02-10 13:54:32,139 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/recovered.edits] 2023-02-10 13:54:32,139 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/recovered.edits] 2023-02-10 13:54:32,142 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a13923017bea4c5386a896c8534246c1 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf1/a13923017bea4c5386a896c8534246c1 2023-02-10 13:54:32,142 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/071e19d3c03743e1ab5fd35170251d29 2023-02-10 13:54:32,143 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/c22d6793c3194a6b895f7c13b53cc37f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf1/c22d6793c3194a6b895f7c13b53cc37f 2023-02-10 13:54:32,144 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/355e2783c02648359e715942fdd8fb7e 2023-02-10 13:54:32,145 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/e48f27de64a64c3b99fbcb2218e0f33c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/cf2/e48f27de64a64c3b99fbcb2218e0f33c 2023-02-10 13:54:32,146 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/3f6e762c937e468ba0d22b2177d9e958 2023-02-10 13:54:32,146 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f84459a5536443068795ed09927e5489 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/cf2/f84459a5536443068795ed09927e5489 2023-02-10 13:54:32,147 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/43be20d10284472d90d6e7347e03ab45 2023-02-10 13:54:32,148 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/recovered.edits/113.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c/recovered.edits/113.seqid 2023-02-10 13:54:32,148 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/45b65037ec8e45a0851d87e4ecdcf932 2023-02-10 13:54:32,149 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/fa1d3c7bf2e8456cfebc77cba079ea0c 2023-02-10 13:54:32,150 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/48196febe2c3448388391680468ad996 2023-02-10 13:54:32,150 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/recovered.edits/113.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d/recovered.edits/113.seqid 2023-02-10 13:54:32,150 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:54:32,151 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/4a536c11616e4cf49789c92f9bae862c 2023-02-10 13:54:32,152 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/9cf83913871e45a4b0c9a978e5ddc959 2023-02-10 13:54:32,154 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/a4ef435470f54f66b97f0a4459dd824c 2023-02-10 13:54:32,155 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf1/bd08426725c14683a929b013b2822c9e 2023-02-10 13:54:32,157 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/212676eabea14142a24806574fb89722 2023-02-10 13:54:32,159 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/4420230fa29f4897b5bf040bb5f3a85b 2023-02-10 13:54:32,160 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/68ce3502c70943f08f4023590e55a6a5 2023-02-10 13:54:32,161 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/8eb040f3ce1e4cd39ccd419705083c98 2023-02-10 13:54:32,163 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/b1738299247e4ebfa823e67260488f85 2023-02-10 13:54:32,164 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/c501090c1e8e4d0e88b16e16c320166a 2023-02-10 13:54:32,165 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/cbd65cc01d084fb79b88a1ee7513b06c 2023-02-10 13:54:32,166 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ddb8723500474913b659135c06d5c549 2023-02-10 13:54:32,167 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/ef8fcc7d47cb431ebaab9b5d1e58e0b8 2023-02-10 13:54:32,168 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/cf2/f41de25e213247808f8461da3586170a 2023-02-10 13:54:32,171 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/recovered.edits/106.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99/recovered.edits/106.seqid 2023-02-10 13:54:32,171 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/61675fc18532b0447c297a024d6bdc99 2023-02-10 13:54:32,171 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-02-10 13:54:32,173 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=84, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,175 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionDeletedRowsDaughter from hbase:meta 2023-02-10 13:54:32,177 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionDeletedRowsDaughter' descriptor. 2023-02-10 13:54:32,178 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=84, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,178 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionDeletedRowsDaughter' from region states. 2023-02-10 13:54:32,178 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037272178"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:32,178 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037272178"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:32,178 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037272178"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:32,180 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:54:32,180 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 61675fc18532b0447c297a024d6bdc99, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1676037268017.61675fc18532b0447c297a024d6bdc99.', STARTKEY => '', ENDKEY => ''}, {ENCODED => fa1d3c7bf2e8456cfebc77cba079ea0c, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1676037270172.fa1d3c7bf2e8456cfebc77cba079ea0c.', STARTKEY => '', ENDKEY => '60'}, {ENCODED => b6601c94d8fe2c319f145aba750fb95d, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1676037270172.b6601c94d8fe2c319f145aba750fb95d.', STARTKEY => '60', ENDKEY => ''}] 2023-02-10 13:54:32,180 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionDeletedRowsDaughter' as deleted. 2023-02-10 13:54:32,180 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037272180"}]},"ts":"9223372036854775807"} 2023-02-10 13:54:32,181 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionDeletedRowsDaughter state from META 2023-02-10 13:54:32,183 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=84, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:54:32,184 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 53 msec 2023-02-10 13:54:32,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-02-10 13:54:32,234 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 84 completed 2023-02-10 13:54:32,249 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=408 (was 403) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:55076 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:46706 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:46730 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:41918 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42002 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:55052 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:41964 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:55054 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:42000 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:55142 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:54986 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:46528 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:55034 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:55046 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-metaLookup-shared--pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:41976 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-113994202_15 at /127.0.0.1:41894 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4b4df50e-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=870 (was 866) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=210 (was 227), ProcessCount=168 (was 168), AvailableMemoryMB=3420 (was 3435) 2023-02-10 13:54:32,263 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=408, OpenFileDescriptor=870, MaxFileDescriptor=60000, SystemLoadAverage=210, ProcessCount=168, AvailableMemoryMB=3420 2023-02-10 13:54:32,263 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:54:32,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:54:32,265 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:54:32,266 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:54:32,268 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:54:32,268 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:54:32,270 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,270 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c empty. 2023-02-10 13:54:32,270 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,270 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-02-10 13:54:32,280 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:32,281 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(7671): creating {ENCODED => c2243fb154cf401741526d8ff8577b4c, NAME => 'testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:54:32,287 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:32,287 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1603): Closing c2243fb154cf401741526d8ff8577b4c, disabling compactions & flushes 2023-02-10 13:54:32,288 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,288 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,288 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. after waiting 0 ms 2023-02-10 13:54:32,288 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,288 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,288 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1557): Region close journal for c2243fb154cf401741526d8ff8577b4c: Waiting for close lock at 1676037272287Disabling compacts and flushes for region at 1676037272287Disabling writes for close at 1676037272288 (+1 ms)Writing region close event to WAL at 1676037272288Closed at 1676037272288 2023-02-10 13:54:32,289 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:54:32,290 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037272290"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037272290"}]},"ts":"1676037272290"} 2023-02-10 13:54:32,291 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:54:32,292 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:54:32,292 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037272292"}]},"ts":"1676037272292"} 2023-02-10 13:54:32,293 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLING in hbase:meta 2023-02-10 13:54:32,297 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:32,297 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:32,297 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:32,297 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:32,297 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:54:32,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, ASSIGN}] 2023-02-10 13:54:32,299 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, ASSIGN 2023-02-10 13:54:32,300 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=false 2023-02-10 13:54:32,450 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:32,453 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:32,453 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037272453"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037272453"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037272453"}]},"ts":"1676037272453"} 2023-02-10 13:54:32,455 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=86, state=RUNNABLE; OpenRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:32,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c2243fb154cf401741526d8ff8577b4c, NAME => 'testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:32,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:32,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:32,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,610 INFO [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,612 DEBUG [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1 2023-02-10 13:54:32,612 DEBUG [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1 2023-02-10 13:54:32,612 INFO [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2243fb154cf401741526d8ff8577b4c columnFamilyName cf1 2023-02-10 13:54:32,613 INFO [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] regionserver.HStore(310): Store=c2243fb154cf401741526d8ff8577b4c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:32,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:54:32,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c2243fb154cf401741526d8ff8577b4c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60827558, jitterRate=-0.09359875321388245}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-10 13:54:32,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c2243fb154cf401741526d8ff8577b4c: Running coprocessor pre-open hook at 1676037272610Writing region info on filesystem at 1676037272610Initializing all the Stores at 1676037272610Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037272610Cleaning up temporary data from old regions at 1676037272614 (+4 ms)Cleaning up detritus from prior splits at 1676037272614Running coprocessor post-open hooks at 1676037272618 (+4 ms)Region opened successfully at 1676037272618 2023-02-10 13:54:32,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c., pid=87, masterSystemTime=1676037272606 2023-02-10 13:54:32,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,620 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:32,620 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037272620"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037272620"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037272620"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037272620"}]},"ts":"1676037272620"} 2023-02-10 13:54:32,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=86 2023-02-10 13:54:32,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=86, state=SUCCESS; OpenRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975 in 166 msec 2023-02-10 13:54:32,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=85 2023-02-10 13:54:32,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=85, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, ASSIGN in 325 msec 2023-02-10 13:54:32,625 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:54:32,625 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037272625"}]},"ts":"1676037272625"} 2023-02-10 13:54:32,626 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLED in hbase:meta 2023-02-10 13:54:32,630 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:54:32,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica in 365 msec 2023-02-10 13:54:32,775 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$12(2441): Client=jenkins//172.31.14.131 modify table testRollbackForSplitTableRegionWithReplica from 'testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'testRollbackForSplitTableRegionWithReplica', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}}, {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-10 13:54:32,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=88, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:54:32,794 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-02-10 13:54:32,795 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-02-10 13:54:32,795 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-02-10 13:54:32,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=89, ppid=88, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica}] 2023-02-10 13:54:32,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, REOPEN/MOVE}] 2023-02-10 13:54:32,810 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, REOPEN/MOVE 2023-02-10 13:54:32,811 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:32,811 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037272811"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037272811"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037272811"}]},"ts":"1676037272811"} 2023-02-10 13:54:32,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=91, ppid=90, state=RUNNABLE; CloseRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:32,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c2243fb154cf401741526d8ff8577b4c, disabling compactions & flushes 2023-02-10 13:54:32,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. after waiting 0 ms 2023-02-10 13:54:32,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-10 13:54:32,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:54:32,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:32,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c2243fb154cf401741526d8ff8577b4c: Waiting for close lock at 1676037272964Running coprocessor pre-close hooks at 1676037272964Disabling compacts and flushes for region at 1676037272964Disabling writes for close at 1676037272966 (+2 ms)Writing region close event to WAL at 1676037272966Running coprocessor post-close hooks at 1676037272969 (+3 ms)Closed at 1676037272969 2023-02-10 13:54:32,969 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c2243fb154cf401741526d8ff8577b4c to self. 2023-02-10 13:54:32,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:32,971 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=CLOSED 2023-02-10 13:54:32,971 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037272971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037272971"}]},"ts":"1676037272971"} 2023-02-10 13:54:32,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=91, resume processing ppid=90 2023-02-10 13:54:32,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, ppid=90, state=SUCCESS; CloseRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975 in 160 msec 2023-02-10 13:54:32,974 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,46307,1676037243975; forceNewPlan=false, retain=true 2023-02-10 13:54:33,125 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:54:33,125 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:33,125 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037273125"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037273125"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037273125"}]},"ts":"1676037273125"} 2023-02-10 13:54:33,127 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=92, ppid=90, state=RUNNABLE; OpenRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:54:33,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:33,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c2243fb154cf401741526d8ff8577b4c, NAME => 'testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:54:33,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:33,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:33,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,283 INFO [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,284 DEBUG [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1 2023-02-10 13:54:33,284 DEBUG [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1 2023-02-10 13:54:33,284 INFO [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c2243fb154cf401741526d8ff8577b4c columnFamilyName cf1 2023-02-10 13:54:33,285 INFO [StoreOpener-c2243fb154cf401741526d8ff8577b4c-1] regionserver.HStore(310): Store=c2243fb154cf401741526d8ff8577b4c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:54:33,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:54:33,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c2243fb154cf401741526d8ff8577b4c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69326942, jitterRate=0.03305193781852722}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-10 13:54:33,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c2243fb154cf401741526d8ff8577b4c: Running coprocessor pre-open hook at 1676037273282Writing region info on filesystem at 1676037273282Initializing all the Stores at 1676037273283 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037273283Cleaning up temporary data from old regions at 1676037273287 (+4 ms)Cleaning up detritus from prior splits at 1676037273287Running coprocessor post-open hooks at 1676037273289 (+2 ms)Region opened successfully at 1676037273289 2023-02-10 13:54:33,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c., pid=92, masterSystemTime=1676037273278 2023-02-10 13:54:33,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:33,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:54:33,291 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:54:33,292 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037273291"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037273291"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037273291"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037273291"}]},"ts":"1676037273291"} 2023-02-10 13:54:33,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=92, resume processing ppid=90 2023-02-10 13:54:33,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, ppid=90, state=SUCCESS; OpenRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975 in 166 msec 2023-02-10 13:54:33,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=90, resume processing ppid=89 2023-02-10 13:54:33,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=90, ppid=89, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, REOPEN/MOVE in 485 msec 2023-02-10 13:54:33,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=89, resume processing ppid=88 2023-02-10 13:54:33,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, ppid=88, state=SUCCESS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica in 499 msec 2023-02-10 13:54:33,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, ASSIGN}] 2023-02-10 13:54:33,302 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, ASSIGN 2023-02-10 13:54:33,302 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-10 13:54:33,453 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:54:33,454 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:54:33,454 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:54:33,454 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:54:33,454 DEBUG [jenkins-hbase4:38205] balancer.BaseLoadBalancer$Cluster(378): Number of tables=3, number of hosts=1, number of racks=1 2023-02-10 13:54:33,456 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=dd8ff2cf66015ae663ea1192dc3ea15d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:54:33,456 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037273455"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1676037273455"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1676037273455"}]},"ts":"1676037273455"} 2023-02-10 13:54:33,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=94, ppid=93, state=RUNNABLE; OpenRegionProcedure dd8ff2cf66015ae663ea1192dc3ea15d, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:54:33,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:54:33,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => dd8ff2cf66015ae663ea1192dc3ea15d, NAME => 'testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2023-02-10 13:54:33,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:54:33,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica dd8ff2cf66015ae663ea1192dc3ea15d 2023-02-10 13:54:33,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:54:33,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for dd8ff2cf66015ae663ea1192dc3ea15d 2023-02-10 13:54:33,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for dd8ff2cf66015ae663ea1192dc3ea15d 2023-02-10 13:54:33,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:33,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:33,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:33,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:34,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,491 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackForSplitTableRegionWithReplica' 2023-02-10 13:54:35,517 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,717 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:35,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,777 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:54:36,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:36,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,621 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:37,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,022 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:38,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,224 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,725 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:39,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,026 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,627 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:40,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:41,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:42,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:43,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:44,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:45,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:46,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,641 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:47,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:48,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:49,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:50,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:51,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:52,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:53,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:54,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,055 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,456 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:55,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,458 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,658 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:56,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,360 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:57,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,362 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:58,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:54:59,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:00,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,067 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,167 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,768 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:01,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,069 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:02,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,171 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,472 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,572 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,672 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:03,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:04,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,449 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-02-10 13:55:05,449 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-02-10 13:55:05,449 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-02-10 13:55:05,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:05,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,077 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:06,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:07,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,481 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:08,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:09,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,485 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:10,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:11,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:12,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:13,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:14,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,695 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:15,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:16,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:17,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:18,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:19,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:20,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:21,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:22,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,510 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:23,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:24,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,514 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:25,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,516 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,717 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:26,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:27,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:28,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:29,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:30,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:31,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,327 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-10 13:55:32,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:32,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-10 13:55:33,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-10 13:55:33,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-10 13:55:33,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,730 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:33,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-10 13:55:34,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,431 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:34,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:35,001 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:35,002 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:35,003 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:35,003 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:35,003 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:35,005 ERROR [PEWorker-1] assignment.SplitTableRegionProcedure(347): Splitting c2243fb154cf401741526d8ff8577b4c, pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up at org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.createUnassignProcedures(SplitTableRegionProcedure.java:881) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:298) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:94) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1646) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:35,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 exec-time=6 msec 2023-02-10 13:55:35,033 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:35,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:35,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-10 13:55:35,334 INFO [StoreOpener-dd8ff2cf66015ae663ea1192dc3ea15d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region dd8ff2cf66015ae663ea1192dc3ea15d 2023-02-10 13:55:35,337 DEBUG [StoreOpener-dd8ff2cf66015ae663ea1192dc3ea15d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1 2023-02-10 13:55:35,337 DEBUG [StoreOpener-dd8ff2cf66015ae663ea1192dc3ea15d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1 2023-02-10 13:55:35,338 INFO [StoreOpener-dd8ff2cf66015ae663ea1192dc3ea15d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dd8ff2cf66015ae663ea1192dc3ea15d columnFamilyName cf1 2023-02-10 13:55:35,338 INFO [StoreOpener-dd8ff2cf66015ae663ea1192dc3ea15d-1] regionserver.HStore(310): Store=dd8ff2cf66015ae663ea1192dc3ea15d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:35,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened dd8ff2cf66015ae663ea1192dc3ea15d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68443553, jitterRate=0.019888415932655334}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-10 13:55:35,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for dd8ff2cf66015ae663ea1192dc3ea15d: Running coprocessor pre-open hook at 1676037273613Initializing all the Stores at 1676037335333 (+61720 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037335333Running coprocessor post-open hooks at 1676037335340 (+7 ms)Region opened successfully at 1676037335340 2023-02-10 13:55:35,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d., pid=94, masterSystemTime=1676037273610 2023-02-10 13:55:35,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:55:35,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:55:35,341 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=dd8ff2cf66015ae663ea1192dc3ea15d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:35,342 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037335341"},{"qualifier":"server_0001","vlen":31,"tag":[],"timestamp":"1676037335341"},{"qualifier":"serverstartcode_0001","vlen":8,"tag":[],"timestamp":"1676037335341"},{"qualifier":"seqnumDuringOpen_0001","vlen":8,"tag":[],"timestamp":"1676037335341"}]},"ts":"1676037335341"} 2023-02-10 13:55:35,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=94, resume processing ppid=93 2023-02-10 13:55:35,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=93, state=SUCCESS; OpenRegionProcedure dd8ff2cf66015ae663ea1192dc3ea15d, server=jenkins-hbase4.apache.org,40651,1676037244010 in 1 mins, 1.886 sec 2023-02-10 13:55:35,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=88 2023-02-10 13:55:35,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=88, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, ASSIGN in 1 mins, 2.044 sec 2023-02-10 13:55:35,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica in 1 mins, 2.569 sec 2023-02-10 13:55:35,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] ipc.RpcServer(528): (responseTooSlow): {"call":"ModifyTable(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest)","starttimems":"1676037272773","responsesize":"2","method":"ModifyTable","param":"TODO: class org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest","processingtimems":62575,"client":"172.31.14.131:53780","queuetimems":0,"class":"HMaster"} 2023-02-10 13:55:35,348 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcDuplexHandler(156): Unknown callId: 603, skipping over this response of 3 bytes 2023-02-10 13:55:37,253 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(335): procId=95 exception: org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up 2023-02-10 13:55:37,254 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:37,256 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,256 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=96, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-02-10 13:55:37,259 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037337259"}]},"ts":"1676037337259"} 2023-02-10 13:55:37,260 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLING in hbase:meta 2023-02-10 13:55:37,262 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackForSplitTableRegionWithReplica to state=DISABLING 2023-02-10 13:55:37,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, UNASSIGN}, {pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, UNASSIGN}] 2023-02-10 13:55:37,263 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, UNASSIGN 2023-02-10 13:55:37,264 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=dd8ff2cf66015ae663ea1192dc3ea15d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:37,264 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037337263"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1676037337263"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1676037337263"}]},"ts":"1676037337263"} 2023-02-10 13:55:37,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; CloseRegionProcedure dd8ff2cf66015ae663ea1192dc3ea15d, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:37,267 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, UNASSIGN 2023-02-10 13:55:37,267 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:37,267 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037337267"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037337267"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037337267"}]},"ts":"1676037337267"} 2023-02-10 13:55:37,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; CloseRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975}] 2023-02-10 13:55:37,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-02-10 13:55:37,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close dd8ff2cf66015ae663ea1192dc3ea15d 2023-02-10 13:55:37,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing dd8ff2cf66015ae663ea1192dc3ea15d, disabling compactions & flushes 2023-02-10 13:55:37,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:55:37,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:55:37,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. after waiting 0 ms 2023-02-10 13:55:37,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:55:37,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:37,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d. 2023-02-10 13:55:37,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for dd8ff2cf66015ae663ea1192dc3ea15d: Waiting for close lock at 1676037337417Running coprocessor pre-close hooks at 1676037337417Disabling compacts and flushes for region at 1676037337417Disabling writes for close at 1676037337418 (+1 ms)Writing region close event to WAL at 1676037337418Running coprocessor post-close hooks at 1676037337418Closed at 1676037337419 (+1 ms) 2023-02-10 13:55:37,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:55:37,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c2243fb154cf401741526d8ff8577b4c, disabling compactions & flushes 2023-02-10 13:55:37,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:55:37,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:55:37,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. after waiting 0 ms 2023-02-10 13:55:37,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:55:37,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed dd8ff2cf66015ae663ea1192dc3ea15d 2023-02-10 13:55:37,424 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=dd8ff2cf66015ae663ea1192dc3ea15d, regionState=CLOSED 2023-02-10 13:55:37,425 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037337423"},{"qualifier":"state_0001","vlen":6,"tag":[],"timestamp":"1676037337423"}]},"ts":"1676037337423"} 2023-02-10 13:55:37,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-10 13:55:37,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:37,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c. 2023-02-10 13:55:37,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-02-10 13:55:37,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c2243fb154cf401741526d8ff8577b4c: Waiting for close lock at 1676037337421Running coprocessor pre-close hooks at 1676037337421Disabling compacts and flushes for region at 1676037337421Disabling writes for close at 1676037337422 (+1 ms)Writing region close event to WAL at 1676037337423 (+1 ms)Running coprocessor post-close hooks at 1676037337429 (+6 ms)Closed at 1676037337430 (+1 ms) 2023-02-10 13:55:37,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; CloseRegionProcedure dd8ff2cf66015ae663ea1192dc3ea15d, server=jenkins-hbase4.apache.org,40651,1676037244010 in 162 msec 2023-02-10 13:55:37,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:55:37,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d, UNASSIGN in 168 msec 2023-02-10 13:55:37,432 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=c2243fb154cf401741526d8ff8577b4c, regionState=CLOSED 2023-02-10 13:55:37,432 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1676037337432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037337432"}]},"ts":"1676037337432"} 2023-02-10 13:55:37,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-02-10 13:55:37,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; CloseRegionProcedure c2243fb154cf401741526d8ff8577b4c, server=jenkins-hbase4.apache.org,46307,1676037243975 in 165 msec 2023-02-10 13:55:37,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=96 2023-02-10 13:55:37,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=c2243fb154cf401741526d8ff8577b4c, UNASSIGN in 175 msec 2023-02-10 13:55:37,439 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037337439"}]},"ts":"1676037337439"} 2023-02-10 13:55:37,440 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLED in hbase:meta 2023-02-10 13:55:37,441 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackForSplitTableRegionWithReplica to state=DISABLED 2023-02-10 13:55:37,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica in 186 msec 2023-02-10 13:55:37,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-02-10 13:55:37,561 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 96 completed 2023-02-10 13:55:37,561 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,564 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=101, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-02-10 13:55:37,570 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:55:37,571 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/recovered.edits] 2023-02-10 13:55:37,575 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/recovered.edits/7.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c/recovered.edits/7.seqid 2023-02-10 13:55:37,575 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/c2243fb154cf401741526d8ff8577b4c 2023-02-10 13:55:37,575 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-02-10 13:55:37,578 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=101, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,579 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackForSplitTableRegionWithReplica from hbase:meta 2023-02-10 13:55:37,581 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackForSplitTableRegionWithReplica' descriptor. 2023-02-10 13:55:37,582 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=101, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,582 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackForSplitTableRegionWithReplica' from region states. 2023-02-10 13:55:37,582 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037337582"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:37,582 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037337582"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:37,584 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 2 regions from META 2023-02-10 13:55:37,584 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c2243fb154cf401741526d8ff8577b4c, NAME => 'testRollbackForSplitTableRegionWithReplica,,1676037272265.c2243fb154cf401741526d8ff8577b4c.', STARTKEY => '', ENDKEY => ''}, {ENCODED => dd8ff2cf66015ae663ea1192dc3ea15d, NAME => 'testRollbackForSplitTableRegionWithReplica,,1676037272265_0001.dd8ff2cf66015ae663ea1192dc3ea15d.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2023-02-10 13:55:37,584 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackForSplitTableRegionWithReplica' as deleted. 2023-02-10 13:55:37,584 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037337584"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:37,585 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackForSplitTableRegionWithReplica state from META 2023-02-10 13:55:37,588 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=101, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:37,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica in 26 msec 2023-02-10 13:55:37,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-02-10 13:55:37,666 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 101 completed 2023-02-10 13:55:37,679 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=342 (was 408), OpenFileDescriptor=656 (was 870), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=74 (was 210), ProcessCount=168 (was 168), AvailableMemoryMB=3314 (was 3420) 2023-02-10 13:55:37,691 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=343, OpenFileDescriptor=656, MaxFileDescriptor=60000, SystemLoadAverage=74, ProcessCount=168, AvailableMemoryMB=3314 2023-02-10 13:55:37,691 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:37,693 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:55:37,693 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:55:37,694 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:37,695 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:55:37,696 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:55:37,697 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:37,698 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 empty. 2023-02-10 13:55:37,698 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:37,698 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-02-10 13:55:37,708 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/.tabledesc/.tableinfo.0000000001 2023-02-10 13:55:37,709 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(7671): creating {ENCODED => a20e2293e915f3127fc63fb13f6dca94, NAME => 'testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testInvalidSplitKey', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:55:37,716 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(865): Instantiated testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:37,716 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1603): Closing a20e2293e915f3127fc63fb13f6dca94, disabling compactions & flushes 2023-02-10 13:55:37,717 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1625): Closing region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:37,717 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:37,717 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1713): Acquired close lock on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. after waiting 0 ms 2023-02-10 13:55:37,717 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1723): Updates disabled for region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:37,717 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1837): Closed testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:37,717 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1557): Region close journal for a20e2293e915f3127fc63fb13f6dca94: Waiting for close lock at 1676037337716Disabling compacts and flushes for region at 1676037337716Disabling writes for close at 1676037337717 (+1 ms)Writing region close event to WAL at 1676037337717Closed at 1676037337717 2023-02-10 13:55:37,719 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:55:37,720 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1676037337719"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037337719"}]},"ts":"1676037337719"} 2023-02-10 13:55:37,721 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:55:37,722 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:55:37,722 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037337722"}]},"ts":"1676037337722"} 2023-02-10 13:55:37,723 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLING in hbase:meta 2023-02-10 13:55:37,725 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:55:37,726 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:55:37,726 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:55:37,726 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:55:37,726 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:55:37,726 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, ASSIGN}] 2023-02-10 13:55:37,728 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, ASSIGN 2023-02-10 13:55:37,728 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:37,878 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:55:37,879 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=a20e2293e915f3127fc63fb13f6dca94, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:37,880 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1676037337879"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037337879"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037337879"}]},"ts":"1676037337879"} 2023-02-10 13:55:37,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=103, state=RUNNABLE; OpenRegionProcedure a20e2293e915f3127fc63fb13f6dca94, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:38,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a20e2293e915f3127fc63fb13f6dca94, NAME => 'testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:55:38,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:38,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testInvalidSplitKey a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:38,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,038 INFO [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,039 DEBUG [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1 2023-02-10 13:55:38,039 DEBUG [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1 2023-02-10 13:55:38,039 INFO [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a20e2293e915f3127fc63fb13f6dca94 columnFamilyName cf1 2023-02-10 13:55:38,040 INFO [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] regionserver.HStore(310): Store=a20e2293e915f3127fc63fb13f6dca94/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:38,040 INFO [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,041 DEBUG [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2 2023-02-10 13:55:38,041 DEBUG [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2 2023-02-10 13:55:38,042 INFO [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a20e2293e915f3127fc63fb13f6dca94 columnFamilyName cf2 2023-02-10 13:55:38,042 INFO [StoreOpener-a20e2293e915f3127fc63fb13f6dca94-1] regionserver.HStore(310): Store=a20e2293e915f3127fc63fb13f6dca94/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:38,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testInvalidSplitKey descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:38,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:38,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:55:38,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a20e2293e915f3127fc63fb13f6dca94; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74514292, jitterRate=0.11034947633743286}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:38,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a20e2293e915f3127fc63fb13f6dca94: Running coprocessor pre-open hook at 1676037338037Writing region info on filesystem at 1676037338037Initializing all the Stores at 1676037338037Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037338037Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037338038 (+1 ms)Cleaning up temporary data from old regions at 1676037338044 (+6 ms)Cleaning up detritus from prior splits at 1676037338044Running coprocessor post-open hooks at 1676037338047 (+3 ms)Region opened successfully at 1676037338047 2023-02-10 13:55:38,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94., pid=104, masterSystemTime=1676037338033 2023-02-10 13:55:38,050 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,050 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=a20e2293e915f3127fc63fb13f6dca94, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,050 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1676037338050"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037338050"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037338050"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037338050"}]},"ts":"1676037338050"} 2023-02-10 13:55:38,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=103 2023-02-10 13:55:38,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=103, state=SUCCESS; OpenRegionProcedure a20e2293e915f3127fc63fb13f6dca94, server=jenkins-hbase4.apache.org,40651,1676037244010 in 170 msec 2023-02-10 13:55:38,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=102 2023-02-10 13:55:38,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=102, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, ASSIGN in 327 msec 2023-02-10 13:55:38,055 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:55:38,055 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037338055"}]},"ts":"1676037338055"} 2023-02-10 13:55:38,056 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLED in hbase:meta 2023-02-10 13:55:38,059 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:55:38,060 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey in 366 msec 2023-02-10 13:55:38,202 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:38,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-10 13:55:38,204 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-10 13:55:38,204 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,204 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-10 13:55:38,204 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-10 13:55:38,205 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,205 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,206 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,206 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,206 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,206 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,206 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,206 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:38,208 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,208 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. started... 2023-02-10 13:55:38,209 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:55:38,209 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,209 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,226 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/a2cd344ef47c4427ba5cb3e4600a56f6 2023-02-10 13:55:38,241 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/97320d6e5257453e9e62b5c2fe062f90 2023-02-10 13:55:38,245 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/a2cd344ef47c4427ba5cb3e4600a56f6 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a2cd344ef47c4427ba5cb3e4600a56f6 2023-02-10 13:55:38,249 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a2cd344ef47c4427ba5cb3e4600a56f6, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:38,250 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/97320d6e5257453e9e62b5c2fe062f90 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/97320d6e5257453e9e62b5c2fe062f90 2023-02-10 13:55:38,253 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/97320d6e5257453e9e62b5c2fe062f90, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:38,254 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 45ms, sequenceid=6, compaction requested=false 2023-02-10 13:55:38,254 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testInvalidSplitKey' 2023-02-10 13:55:38,255 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:38,255 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,255 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:38,255 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:38,255 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,255 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,255 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,256 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,256 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,257 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,257 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-10 13:55:38,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ceaf896[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:38,258 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-10 13:55:38,259 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,260 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,260 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,260 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,261 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:38,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,261 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:38,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,261 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:38,262 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,262 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,262 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,263 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,263 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:38,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-10 13:55:38,266 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-10 13:55:38,266 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:38,266 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:38,266 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-10 13:55:38,266 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:38,268 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,269 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,269 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,269 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,270 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,270 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,271 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,272 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,272 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,275 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,275 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-10 13:55:38,275 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:38,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:38,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,275 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,276 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,276 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,276 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,276 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,376 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:38,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:38,390 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-10 13:55:38,392 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-10 13:55:38,392 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,392 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-10 13:55:38,392 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-10 13:55:38,392 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,392 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,394 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,394 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,395 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,396 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:38,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:38,397 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,397 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. started... 2023-02-10 13:55:38,397 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:38,402 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,402 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,403 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,403 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,403 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,403 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,417 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 2023-02-10 13:55:38,434 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/e5feb5807d5f46ec8ffa3b8816806842 2023-02-10 13:55:38,439 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 2023-02-10 13:55:38,443 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a88bb7f3e63b437a91eb3885ed89a8f2, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:38,444 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/e5feb5807d5f46ec8ffa3b8816806842 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/e5feb5807d5f46ec8ffa3b8816806842 2023-02-10 13:55:38,449 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/e5feb5807d5f46ec8ffa3b8816806842, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:38,450 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 53ms, sequenceid=19, compaction requested=false 2023-02-10 13:55:38,450 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:38,450 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,450 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,452 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,452 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,453 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,453 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-10 13:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db8f0fa[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:38,455 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-10 13:55:38,455 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,456 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,456 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:38,456 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,457 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,459 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:38,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-10 13:55:38,463 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-10 13:55:38,463 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:38,463 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:38,463 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-10 13:55:38,463 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,465 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,465 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,466 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,467 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,468 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,468 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,469 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,471 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-10 13:55:38,472 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:38,471 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,471 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:38,472 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,573 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:38,587 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:38,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:38,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-10 13:55:38,588 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-10 13:55:38,588 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,588 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-10 13:55:38,589 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-10 13:55:38,589 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,589 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,591 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,592 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,592 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:38,594 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,594 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. started... 2023-02-10 13:55:38,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,594 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,594 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,596 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,598 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,598 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,598 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,605 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/22720629be96420fa973e42974e3e390 2023-02-10 13:55:38,626 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/254e4b46699e4ed49744e054cf4dcca9 2023-02-10 13:55:38,631 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/22720629be96420fa973e42974e3e390 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/22720629be96420fa973e42974e3e390 2023-02-10 13:55:38,635 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/22720629be96420fa973e42974e3e390, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:38,635 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/254e4b46699e4ed49744e054cf4dcca9 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/254e4b46699e4ed49744e054cf4dcca9 2023-02-10 13:55:38,639 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/254e4b46699e4ed49744e054cf4dcca9, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:38,640 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 46ms, sequenceid=32, compaction requested=true 2023-02-10 13:55:38,640 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:38,640 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,640 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:38,640 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:38,640 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,640 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,640 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,642 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,642 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,642 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,642 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-10 13:55:38,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316f59c[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:38,644 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-10 13:55:38,644 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,646 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,646 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,646 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,647 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,647 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:38,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,647 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,648 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,649 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,649 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,649 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,649 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:38,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-10 13:55:38,652 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-10 13:55:38,652 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:38,652 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:38,652 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-10 13:55:38,652 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:38,654 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,654 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,654 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,654 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,654 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,655 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,655 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,656 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,657 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,658 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,658 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,663 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,663 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,663 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,663 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,664 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,664 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-10 13:55:38,664 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,664 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:38,665 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,765 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:38,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:38,781 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:38,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:38,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-10 13:55:38,785 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-10 13:55:38,785 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,785 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-10 13:55:38,785 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-10 13:55:38,785 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,785 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,786 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,786 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,786 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,786 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,787 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,787 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,789 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,790 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,791 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,793 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,793 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,793 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,793 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,793 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,796 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:38,796 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:38,796 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,796 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. started... 2023-02-10 13:55:38,796 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:38,813 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/c9395eb33192497f862ba5b5a7bc2844 2023-02-10 13:55:38,830 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/ec9336f8e8794ce2ae935caae5eb47dc 2023-02-10 13:55:38,835 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/c9395eb33192497f862ba5b5a7bc2844 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c9395eb33192497f862ba5b5a7bc2844 2023-02-10 13:55:38,839 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c9395eb33192497f862ba5b5a7bc2844, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:38,840 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/ec9336f8e8794ce2ae935caae5eb47dc as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/ec9336f8e8794ce2ae935caae5eb47dc 2023-02-10 13:55:38,844 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/ec9336f8e8794ce2ae935caae5eb47dc, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:38,845 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 49ms, sequenceid=45, compaction requested=true 2023-02-10 13:55:38,845 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:38,845 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,845 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,847 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,847 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-10 13:55:38,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f01f375[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:38,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-10 13:55:38,849 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,850 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,850 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,850 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,851 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,851 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:38,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:38,852 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,853 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,853 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,853 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,854 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-10 13:55:38,857 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-10 13:55:38,857 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:38,857 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:38,858 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-10 13:55:38,858 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,859 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,860 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,861 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,862 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:38,862 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:38,863 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,865 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,865 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,866 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:38,868 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,865 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,872 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,873 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-10 13:55:38,873 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:38,873 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:38,873 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:38,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,865 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,865 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:38,880 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,881 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:38,881 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,881 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,973 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:38,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:38,985 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:38,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:38,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-10 13:55:38,987 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-10 13:55:38,987 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,987 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-10 13:55:38,987 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-10 13:55:38,987 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,987 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,988 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,988 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:38,989 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:38,989 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:38,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:38,991 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:38,991 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. started... 2023-02-10 13:55:38,991 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:39,020 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/f0b326c462844790840159c8596f9334 2023-02-10 13:55:39,031 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/f0aea36697bc4a13a0b245465e0bf378 2023-02-10 13:55:39,034 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/f0b326c462844790840159c8596f9334 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/f0b326c462844790840159c8596f9334 2023-02-10 13:55:39,038 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/f0b326c462844790840159c8596f9334, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:39,039 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/f0aea36697bc4a13a0b245465e0bf378 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/f0aea36697bc4a13a0b245465e0bf378 2023-02-10 13:55:39,042 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/f0aea36697bc4a13a0b245465e0bf378, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:39,043 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 52ms, sequenceid=58, compaction requested=true 2023-02-10 13:55:39,043 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:39,043 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:39,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:39,045 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,045 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:39,045 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,045 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:39,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-10 13:55:39,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74b4984b[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:39,047 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-10 13:55:39,047 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,048 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:39,048 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:39,049 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:39,054 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:39,054 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,054 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,057 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:39,057 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:39,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-10 13:55:39,060 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-10 13:55:39,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:39,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:39,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-10 13:55:39,060 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:39,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:39,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,062 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,062 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,062 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,062 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,063 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,064 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:39,064 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,065 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,068 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-10 13:55:39,069 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:39,068 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,169 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:39,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:39,181 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:39,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:39,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-10 13:55:39,182 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-10 13:55:39,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:39,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-10 13:55:39,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-10 13:55:39,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,183 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,183 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,184 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,184 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:39,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-10 13:55:39,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:39,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:39,185 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:39,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-10 13:55:39,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:39,186 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:39,186 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,186 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. started... 2023-02-10 13:55:39,187 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:39,187 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:39,195 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/c138dbd9b759471eba5dde8957118bbb 2023-02-10 13:55:39,205 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/b0a10fdc6896483db8cec6c4a2521dd3 2023-02-10 13:55:39,209 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/c138dbd9b759471eba5dde8957118bbb as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c138dbd9b759471eba5dde8957118bbb 2023-02-10 13:55:39,213 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c138dbd9b759471eba5dde8957118bbb, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:39,213 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/b0a10fdc6896483db8cec6c4a2521dd3 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/b0a10fdc6896483db8cec6c4a2521dd3 2023-02-10 13:55:39,217 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/b0a10fdc6896483db8cec6c4a2521dd3, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:39,218 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 31ms, sequenceid=71, compaction requested=true 2023-02-10 13:55:39,218 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:39,218 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-10 13:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-10 13:55:39,219 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:39,220 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,220 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:39,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-10 13:55:39,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e619907[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:39,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-10 13:55:39,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,228 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,228 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,228 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:39,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,229 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,229 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:39,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-10 13:55:39,229 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:39,230 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:39,230 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,230 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:39,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-10 13:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:39,231 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-10 13:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-10 13:55:39,234 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-10 13:55:39,234 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:39,234 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:39,234 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-10 13:55:39,234 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:39,235 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,235 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:39,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,235 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,235 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,235 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:39,236 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,236 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,236 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:39,237 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:39,238 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-10 13:55:39,238 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,239 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,242 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:39,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,242 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-10 13:55:39,242 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:39,242 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-10 13:55:39,243 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:39,243 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,243 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,243 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:39,243 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-10 13:55:39,243 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-10 13:55:39,343 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:39,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:39,351 INFO [Listener at localhost/43477] assignment.SplitTableRegionProcedure(206): testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. splitKey isn't explicitly specified, will try to find a best split key from RS jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,353 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.StoreUtils(129): cannot split hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 because midkey is the same as first or last row 2023-02-10 13:55:39,353 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.StoreUtils(129): cannot split hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/e5feb5807d5f46ec8ffa3b8816806842 because midkey is the same as first or last row 2023-02-10 13:55:39,353 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegion(2744): Flushing a20e2293e915f3127fc63fb13f6dca94 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:55:39,367 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/8bf6ba8b330e48ae806dbeadc2543e86 2023-02-10 13:55:39,379 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/48b47b876d9445e286165acbdd1ddd5f 2023-02-10 13:55:39,383 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf1/8bf6ba8b330e48ae806dbeadc2543e86 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/8bf6ba8b330e48ae806dbeadc2543e86 2023-02-10 13:55:39,386 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/8bf6ba8b330e48ae806dbeadc2543e86, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:39,386 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/.tmp/cf2/48b47b876d9445e286165acbdd1ddd5f as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/48b47b876d9445e286165acbdd1ddd5f 2023-02-10 13:55:39,390 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/48b47b876d9445e286165acbdd1ddd5f, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:39,390 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for a20e2293e915f3127fc63fb13f6dca94 in 37ms, sequenceid=82, compaction requested=true 2023-02-10 13:55:39,391 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.HRegion(2445): Flush status journal for a20e2293e915f3127fc63fb13f6dca94: 2023-02-10 13:55:39,391 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.StoreUtils(129): cannot split hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 because midkey is the same as first or last row 2023-02-10 13:55:39,391 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=40651] regionserver.StoreUtils(129): cannot split hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/e5feb5807d5f46ec8ffa3b8816806842 because midkey is the same as first or last row 2023-02-10 13:55:39,392 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,392 DEBUG [Listener at localhost/43477] assignment.TestSplitTableRegionProcedure(438): Expected Split procedure construction failure: Region not splittable because bestSplitPoint = null, maybe table is too small for auto split. For force split, try specifying split row 2023-02-10 13:55:39,392 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:39,394 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testInvalidSplitKey 2023-02-10 13:55:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testInvalidSplitKey 2023-02-10 13:55:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-10 13:55:39,397 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037339397"}]},"ts":"1676037339397"} 2023-02-10 13:55:39,398 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLING in hbase:meta 2023-02-10 13:55:39,399 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testInvalidSplitKey to state=DISABLING 2023-02-10 13:55:39,400 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, UNASSIGN}] 2023-02-10 13:55:39,400 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, UNASSIGN 2023-02-10 13:55:39,401 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=a20e2293e915f3127fc63fb13f6dca94, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:39,401 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1676037339401"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037339401"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037339401"}]},"ts":"1676037339401"} 2023-02-10 13:55:39,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=107, ppid=106, state=RUNNABLE; CloseRegionProcedure a20e2293e915f3127fc63fb13f6dca94, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:39,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-10 13:55:39,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:39,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a20e2293e915f3127fc63fb13f6dca94, disabling compactions & flushes 2023-02-10 13:55:39,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. after waiting 0 ms 2023-02-10 13:55:39,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:55:39,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:39,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94. 2023-02-10 13:55:39,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a20e2293e915f3127fc63fb13f6dca94: Waiting for close lock at 1676037339554Running coprocessor pre-close hooks at 1676037339554Disabling compacts and flushes for region at 1676037339554Disabling writes for close at 1676037339554Writing region close event to WAL at 1676037339555 (+1 ms)Running coprocessor post-close hooks at 1676037339558 (+3 ms)Closed at 1676037339558 2023-02-10 13:55:39,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:39,560 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=a20e2293e915f3127fc63fb13f6dca94, regionState=CLOSED 2023-02-10 13:55:39,560 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1676037339560"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037339560"}]},"ts":"1676037339560"} 2023-02-10 13:55:39,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=106 2023-02-10 13:55:39,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=106, state=SUCCESS; CloseRegionProcedure a20e2293e915f3127fc63fb13f6dca94, server=jenkins-hbase4.apache.org,40651,1676037244010 in 159 msec 2023-02-10 13:55:39,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-02-10 13:55:39,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=a20e2293e915f3127fc63fb13f6dca94, UNASSIGN in 162 msec 2023-02-10 13:55:39,564 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037339564"}]},"ts":"1676037339564"} 2023-02-10 13:55:39,565 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLED in hbase:meta 2023-02-10 13:55:39,566 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testInvalidSplitKey to state=DISABLED 2023-02-10 13:55:39,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey in 173 msec 2023-02-10 13:55:39,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-10 13:55:39,699 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testInvalidSplitKey, procId: 105 completed 2023-02-10 13:55:39,699 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testInvalidSplitKey 2023-02-10 13:55:39,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,702 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,702 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=108, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-02-10 13:55:39,705 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:39,707 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/recovered.edits] 2023-02-10 13:55:39,710 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/22720629be96420fa973e42974e3e390 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/22720629be96420fa973e42974e3e390 2023-02-10 13:55:39,711 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/8bf6ba8b330e48ae806dbeadc2543e86 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/8bf6ba8b330e48ae806dbeadc2543e86 2023-02-10 13:55:39,712 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a2cd344ef47c4427ba5cb3e4600a56f6 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a2cd344ef47c4427ba5cb3e4600a56f6 2023-02-10 13:55:39,713 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/a88bb7f3e63b437a91eb3885ed89a8f2 2023-02-10 13:55:39,714 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c138dbd9b759471eba5dde8957118bbb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c138dbd9b759471eba5dde8957118bbb 2023-02-10 13:55:39,715 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c9395eb33192497f862ba5b5a7bc2844 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/c9395eb33192497f862ba5b5a7bc2844 2023-02-10 13:55:39,716 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/f0b326c462844790840159c8596f9334 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf1/f0b326c462844790840159c8596f9334 2023-02-10 13:55:39,719 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/254e4b46699e4ed49744e054cf4dcca9 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/254e4b46699e4ed49744e054cf4dcca9 2023-02-10 13:55:39,720 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/48b47b876d9445e286165acbdd1ddd5f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/48b47b876d9445e286165acbdd1ddd5f 2023-02-10 13:55:39,721 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/97320d6e5257453e9e62b5c2fe062f90 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/97320d6e5257453e9e62b5c2fe062f90 2023-02-10 13:55:39,722 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/b0a10fdc6896483db8cec6c4a2521dd3 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/b0a10fdc6896483db8cec6c4a2521dd3 2023-02-10 13:55:39,723 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/e5feb5807d5f46ec8ffa3b8816806842 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/e5feb5807d5f46ec8ffa3b8816806842 2023-02-10 13:55:39,724 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/ec9336f8e8794ce2ae935caae5eb47dc to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/ec9336f8e8794ce2ae935caae5eb47dc 2023-02-10 13:55:39,725 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/f0aea36697bc4a13a0b245465e0bf378 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/cf2/f0aea36697bc4a13a0b245465e0bf378 2023-02-10 13:55:39,727 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/recovered.edits/85.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94/recovered.edits/85.seqid 2023-02-10 13:55:39,728 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testInvalidSplitKey/a20e2293e915f3127fc63fb13f6dca94 2023-02-10 13:55:39,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-02-10 13:55:39,729 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=108, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,731 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testInvalidSplitKey from hbase:meta 2023-02-10 13:55:39,734 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testInvalidSplitKey' descriptor. 2023-02-10 13:55:39,735 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=108, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,735 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testInvalidSplitKey' from region states. 2023-02-10 13:55:39,735 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037339735"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:39,737 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-02-10 13:55:39,737 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a20e2293e915f3127fc63fb13f6dca94, NAME => 'testInvalidSplitKey,,1676037337693.a20e2293e915f3127fc63fb13f6dca94.', STARTKEY => '', ENDKEY => ''}] 2023-02-10 13:55:39,737 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testInvalidSplitKey' as deleted. 2023-02-10 13:55:39,737 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037339737"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:39,738 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testInvalidSplitKey state from META 2023-02-10 13:55:39,740 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=108, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:39,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey in 41 msec 2023-02-10 13:55:39,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-02-10 13:55:39,803 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testInvalidSplitKey, procId: 108 completed 2023-02-10 13:55:39,817 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=365 (was 343) - Thread LEAK? -, OpenFileDescriptor=742 (was 656) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=74 (was 74), ProcessCount=168 (was 168), AvailableMemoryMB=3205 (was 3314) 2023-02-10 13:55:39,830 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=365, OpenFileDescriptor=742, MaxFileDescriptor=60000, SystemLoadAverage=74, ProcessCount=168, AvailableMemoryMB=3204 2023-02-10 13:55:39,830 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:39,832 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:55:39,832 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:55:39,833 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:39,835 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:55:39,835 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:55:39,836 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:39,837 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 empty. 2023-02-10 13:55:39,837 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:39,837 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-02-10 13:55:39,846 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-02-10 13:55:39,847 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(7671): creating {ENCODED => d1e6e5a45d298475173074736dd3b162, NAME => 'testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRecoveryAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:55:39,853 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:39,853 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1603): Closing d1e6e5a45d298475173074736dd3b162, disabling compactions & flushes 2023-02-10 13:55:39,853 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:39,853 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:39,853 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. after waiting 0 ms 2023-02-10 13:55:39,853 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:39,853 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:39,853 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1557): Region close journal for d1e6e5a45d298475173074736dd3b162: Waiting for close lock at 1676037339853Disabling compacts and flushes for region at 1676037339853Disabling writes for close at 1676037339853Writing region close event to WAL at 1676037339853Closed at 1676037339853 2023-02-10 13:55:39,855 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:55:39,855 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037339855"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037339855"}]},"ts":"1676037339855"} 2023-02-10 13:55:39,856 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:55:39,857 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:55:39,857 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037339857"}]},"ts":"1676037339857"} 2023-02-10 13:55:39,858 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLING in hbase:meta 2023-02-10 13:55:39,860 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:55:39,861 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:55:39,861 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:55:39,861 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:55:39,861 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:55:39,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, ASSIGN}] 2023-02-10 13:55:39,862 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, ASSIGN 2023-02-10 13:55:39,863 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:40,013 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:55:40,014 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=d1e6e5a45d298475173074736dd3b162, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,014 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037340014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037340014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037340014"}]},"ts":"1676037340014"} 2023-02-10 13:55:40,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=111, ppid=110, state=RUNNABLE; OpenRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:40,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d1e6e5a45d298475173074736dd3b162, NAME => 'testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:55:40,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,172 INFO [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,173 DEBUG [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1 2023-02-10 13:55:40,173 DEBUG [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1 2023-02-10 13:55:40,174 INFO [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1e6e5a45d298475173074736dd3b162 columnFamilyName cf1 2023-02-10 13:55:40,174 INFO [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] regionserver.HStore(310): Store=d1e6e5a45d298475173074736dd3b162/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:40,175 INFO [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,176 DEBUG [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2 2023-02-10 13:55:40,176 DEBUG [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2 2023-02-10 13:55:40,176 INFO [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d1e6e5a45d298475173074736dd3b162 columnFamilyName cf2 2023-02-10 13:55:40,177 INFO [StoreOpener-d1e6e5a45d298475173074736dd3b162-1] regionserver.HStore(310): Store=d1e6e5a45d298475173074736dd3b162/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:40,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:40,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:40,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:55:40,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d1e6e5a45d298475173074736dd3b162; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59798694, jitterRate=-0.10893002152442932}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:40,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d1e6e5a45d298475173074736dd3b162: Running coprocessor pre-open hook at 1676037340171Writing region info on filesystem at 1676037340171Initializing all the Stores at 1676037340172 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037340172Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037340172Cleaning up temporary data from old regions at 1676037340189 (+17 ms)Cleaning up detritus from prior splits at 1676037340190 (+1 ms)Running coprocessor post-open hooks at 1676037340195 (+5 ms)Region opened successfully at 1676037340195 2023-02-10 13:55:40,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162., pid=111, masterSystemTime=1676037340167 2023-02-10 13:55:40,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,198 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=d1e6e5a45d298475173074736dd3b162, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,198 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037340198"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037340198"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037340198"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037340198"}]},"ts":"1676037340198"} 2023-02-10 13:55:40,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=110 2023-02-10 13:55:40,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=110, state=SUCCESS; OpenRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 in 184 msec 2023-02-10 13:55:40,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=110, resume processing ppid=109 2023-02-10 13:55:40,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=109, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, ASSIGN in 340 msec 2023-02-10 13:55:40,203 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:55:40,204 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037340204"}]},"ts":"1676037340204"} 2023-02-10 13:55:40,205 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLED in hbase:meta 2023-02-10 13:55:40,208 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:55:40,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution in 376 msec 2023-02-10 13:55:40,341 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:40,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-10 13:55:40,342 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,342 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,342 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-10 13:55:40,342 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:55:40,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,344 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,344 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,344 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,344 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,345 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:40,346 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,346 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. started... 2023-02-10 13:55:40,347 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:55:40,348 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,348 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,348 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,356 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/03d29cf5d6b74147a22387e625b3b9ce 2023-02-10 13:55:40,367 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/51814fa3b178448a9c8139bccaa35e9d 2023-02-10 13:55:40,371 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/03d29cf5d6b74147a22387e625b3b9ce as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce 2023-02-10 13:55:40,374 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:40,375 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/51814fa3b178448a9c8139bccaa35e9d as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d 2023-02-10 13:55:40,378 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:40,378 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 31ms, sequenceid=6, compaction requested=false 2023-02-10 13:55:40,378 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,379 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d1e6e5a45d298475173074736dd3b162: 2023-02-10 13:55:40,379 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,379 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:40,379 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:40,379 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,379 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,379 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,380 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,381 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,381 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,381 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-10 13:55:40,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a6f3542[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:40,383 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:55:40,383 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:40,386 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,386 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:40,387 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,387 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,387 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,388 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:40,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:55:40,391 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-10 13:55:40,391 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:40,391 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:40,391 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,391 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:40,392 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,392 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,392 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,392 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,393 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,392 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,393 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,394 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,394 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,395 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,395 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,396 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,398 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,399 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,399 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,399 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,399 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,500 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:40,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:40,512 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:40,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:40,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-10 13:55:40,513 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,513 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,513 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-10 13:55:40,513 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:55:40,514 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,514 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,515 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,515 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,515 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,515 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,516 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,517 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:40,518 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,518 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. started... 2023-02-10 13:55:40,518 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,526 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/9678d2d995fa4e01add3adf639c1a8b2 2023-02-10 13:55:40,536 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 2023-02-10 13:55:40,539 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/9678d2d995fa4e01add3adf639c1a8b2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 2023-02-10 13:55:40,543 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:40,543 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 2023-02-10 13:55:40,547 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:40,547 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 29ms, sequenceid=19, compaction requested=false 2023-02-10 13:55:40,548 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d1e6e5a45d298475173074736dd3b162: 2023-02-10 13:55:40,548 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,548 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,558 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,558 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,558 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,558 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-10 13:55:40,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a95e121[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:40,561 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:55:40,561 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,562 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,562 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,563 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,563 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:40,562 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,564 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,564 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,565 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:40,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:55:40,567 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-10 13:55:40,567 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:40,567 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:40,567 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,568 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,571 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,571 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,572 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,573 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,574 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,574 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,577 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,577 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,577 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:40,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,577 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,578 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,578 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,578 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,578 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,578 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,678 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:40,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:40,689 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-10 13:55:40,691 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,691 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,691 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-10 13:55:40,691 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:55:40,691 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,691 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,692 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,692 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,692 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,692 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,693 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,693 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,695 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,695 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. started... 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,695 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,696 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,697 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,697 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,697 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,697 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,706 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 2023-02-10 13:55:40,717 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/d5219efa2d734ef1883c1c9a0b693bfa 2023-02-10 13:55:40,721 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 2023-02-10 13:55:40,724 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:40,725 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/d5219efa2d734ef1883c1c9a0b693bfa as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa 2023-02-10 13:55:40,728 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:40,729 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 34ms, sequenceid=32, compaction requested=true 2023-02-10 13:55:40,729 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d1e6e5a45d298475173074736dd3b162: 2023-02-10 13:55:40,729 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:40,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:40,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,732 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,732 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,732 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,732 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-10 13:55:40,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab34ef8[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:40,734 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:55:40,734 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,735 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,735 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:40,735 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,736 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,737 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,737 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,738 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:55:40,740 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-10 13:55:40,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:40,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:40,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,740 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:40,742 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,742 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,742 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,743 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,743 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,744 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,745 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,745 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,746 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,751 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,751 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,751 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,752 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:40,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,752 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,751 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,752 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,752 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,852 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:40,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:40,863 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:40,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:40,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-10 13:55:40,865 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,865 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,865 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-10 13:55:40,865 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:55:40,865 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,865 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,866 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,866 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,866 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,866 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,867 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,867 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,868 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:40,869 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:40,869 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:40,869 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:40,869 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,869 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. started... 2023-02-10 13:55:40,869 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:40,871 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,872 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,877 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/b5cfda0e01774a8698fa2182026bd613 2023-02-10 13:55:40,888 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 2023-02-10 13:55:40,892 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/b5cfda0e01774a8698fa2182026bd613 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 2023-02-10 13:55:40,896 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:40,897 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 2023-02-10 13:55:40,900 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:40,901 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 32ms, sequenceid=45, compaction requested=true 2023-02-10 13:55:40,901 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d1e6e5a45d298475173074736dd3b162: 2023-02-10 13:55:40,901 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:40,903 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,903 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,903 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,903 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:40,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-10 13:55:40,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8085508[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:40,905 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:55:40,905 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,906 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,906 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,907 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,910 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,910 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:40,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:55:40,913 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-10 13:55:40,913 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:40,913 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:40,913 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-10 13:55:40,913 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:40,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,915 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,915 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:40,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,916 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,916 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:40,918 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:40,918 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,918 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,921 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,921 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,921 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,921 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:40,921 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,921 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:40,921 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:40,921 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:40,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,921 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:40,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:40,922 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:40,922 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:40,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:40,924 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,022 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:41,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:41,033 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:41,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:41,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-10 13:55:41,034 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:41,034 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,034 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-10 13:55:41,034 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:55:41,035 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,035 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,037 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,037 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,037 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,037 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,037 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,038 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:41,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:41,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:41,040 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:41,040 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:41,040 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:41,040 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:41,040 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. started... 2023-02-10 13:55:41,040 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:41,048 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/479381402ff6407ba4e10af6ae2160cf 2023-02-10 13:55:41,060 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/1269f987775e47b892c4d726ac112b4c 2023-02-10 13:55:41,064 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/479381402ff6407ba4e10af6ae2160cf as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf 2023-02-10 13:55:41,068 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:41,069 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/1269f987775e47b892c4d726ac112b4c as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c 2023-02-10 13:55:41,074 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:41,075 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 35ms, sequenceid=58, compaction requested=true 2023-02-10 13:55:41,075 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d1e6e5a45d298475173074736dd3b162: 2023-02-10 13:55:41,075 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:41,075 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:41,075 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:41,075 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,075 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:41,075 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:41,081 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,081 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:41,082 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,082 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:41,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-10 13:55:41,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1ea46e[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:41,085 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:55:41,085 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:41,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,086 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:41,086 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:41,087 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,087 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:41,087 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:41,088 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,088 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:41,088 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:41,088 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:41,089 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:41,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:55:41,092 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-10 13:55:41,092 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:41,092 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:41,092 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,092 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:41,097 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:41,097 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,097 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,098 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,098 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,099 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,099 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,101 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:41,101 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,102 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,105 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,105 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:41,105 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:41,105 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,106 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,206 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:41,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:41,217 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:41,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:41,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-10 13:55:41,219 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:41,219 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,219 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-10 13:55:41,219 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-10 13:55:41,219 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,219 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,221 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,221 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,221 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:41,222 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-10 13:55:41,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:41,223 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:41,223 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:41,223 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. started... 2023-02-10 13:55:41,223 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:41,224 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,230 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,230 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,231 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:41,231 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,231 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:41,231 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/80eaed8e110641358db46bf070937cd9 2023-02-10 13:55:41,241 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/4ccd28f5ae02471789d62512f3c7aed2 2023-02-10 13:55:41,245 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/80eaed8e110641358db46bf070937cd9 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 2023-02-10 13:55:41,248 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:41,249 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/4ccd28f5ae02471789d62512f3c7aed2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 2023-02-10 13:55:41,251 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:41,252 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 29ms, sequenceid=71, compaction requested=true 2023-02-10 13:55:41,252 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d1e6e5a45d298475173074736dd3b162: 2023-02-10 13:55:41,252 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-10 13:55:41,253 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-10 13:55:41,254 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,254 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:41,254 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,254 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:41,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-10 13:55:41,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee7dac5[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:41,256 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-10 13:55:41,256 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,257 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:41,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:41,257 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:41,257 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:41,257 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:41,257 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,257 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:41,258 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-10 13:55:41,258 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-10 13:55:41,258 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:41,259 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:41,259 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,259 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:41,260 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-10 13:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:41,260 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-10 13:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:41,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-10 13:55:41,263 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-10 13:55:41,263 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:41,263 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:41,263 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-10 13:55:41,263 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,265 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:41,265 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,265 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,266 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,267 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:41,268 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-10 13:55:41,268 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,273 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:41,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,273 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-10 13:55:41,274 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,273 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:41,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-10 13:55:41,374 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:41,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:41,633 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-02-10 13:55:41,634 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:41,634 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,636 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:41,636 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:41,636 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:41,885 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 0 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,885 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:41,885 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:41,885 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:41,885 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] 2023-02-10 13:55:41,885 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:41,886 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:41,886 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:41,886 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:41,886 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:41,888 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-02-10 13:55:41,888 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:41,889 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:41,889 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:41,889 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:41,890 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:41,890 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:41,890 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:41,890 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:41,891 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:41,891 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:41,891 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:41,891 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:41,892 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:41,892 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:41,892 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:41,892 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:41,893 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:41,893 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:41,893 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:41,893 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:41,893 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:41,894 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:41,894 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:41,894 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:41,894 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:41,895 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:41,895 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:41,896 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:41,896 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:41,896 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:41,896 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:41,897 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:41,897 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:41,897 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:41,897 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,897 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:41,898 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,898 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,898 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,898 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 11 msec 2023-02-10 13:55:41,899 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:41,899 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:41,899 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037371899 2023-02-10 13:55:41,901 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:41,901 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,903 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:41,903 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:41,903 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:41,904 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037401904 2023-02-10 13:55:41,904 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037461904 2023-02-10 13:55:41,904 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:55:41,904 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:41,906 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:41,906 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:41,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:41,906 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:42,154 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 1 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,154 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:42,154 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:42,154 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:42,155 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:42,155 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:42,155 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:42,155 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:42,155 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:42,156 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:42,157 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-10 13:55:42,157 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:42,158 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:42,158 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:42,158 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:42,159 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,159 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:42,159 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,159 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,160 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,160 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:42,160 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,160 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,161 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,161 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:42,161 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:42,162 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,162 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,162 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,162 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:42,163 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,163 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,163 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,164 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:42,164 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,164 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,164 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,165 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,165 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:42,165 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,165 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,166 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,166 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,166 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,166 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:42,167 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,167 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:42,167 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,167 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,167 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,167 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 11 msec 2023-02-10 13:55:42,168 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:42,169 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:42,172 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037372172 2023-02-10 13:55:42,173 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:42,176 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:42,176 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:42,176 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:42,176 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037402176 2023-02-10 13:55:42,176 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037462176 2023-02-10 13:55:42,176 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:55:42,181 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:42,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN}] 2023-02-10 13:55:42,183 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:42,183 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:42,183 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:42,427 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 2 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,427 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:42,427 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:42,427 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:42,427 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:42,428 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:42,428 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:42,428 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:42,428 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:42,429 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:42,430 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-10 13:55:42,434 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:42,435 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:42,435 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:42,435 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:42,435 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,436 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:42,436 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,436 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,436 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,437 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:42,437 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,437 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,437 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,438 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:42,438 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:42,438 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,438 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,438 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,439 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:42,439 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,439 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,439 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,440 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:42,440 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,440 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,440 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,440 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,441 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:42,441 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,441 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,441 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,442 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,442 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,442 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:42,442 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,443 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:42,443 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,443 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,443 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,443 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 14 msec 2023-02-10 13:55:42,444 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:42,444 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:42,451 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037372450 2023-02-10 13:55:42,451 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,451 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:42,454 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:42,454 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:42,455 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:42,455 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037402455 2023-02-10 13:55:42,455 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037462455 2023-02-10 13:55:42,455 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:55:42,455 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN}] 2023-02-10 13:55:42,455 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:42,461 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:42,464 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=d1e6e5a45d298475173074736dd3b162, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:42,464 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037342464"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037342464"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037342464"}]},"ts":"1676037342464"} 2023-02-10 13:55:42,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=113, state=RUNNABLE; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:42,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d1e6e5a45d298475173074736dd3b162, disabling compactions & flushes 2023-02-10 13:55:42,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:42,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:42,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. after waiting 0 ms 2023-02-10 13:55:42,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:42,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d1e6e5a45d298475173074736dd3b162 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:55:42,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/781e33094eeb4a09a7f361994eff3857 2023-02-10 13:55:42,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 2023-02-10 13:55:42,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf1/781e33094eeb4a09a7f361994eff3857 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 2023-02-10 13:55:42,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:42,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/.tmp/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 2023-02-10 13:55:42,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:42,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 35ms, sequenceid=82, compaction requested=true 2023-02-10 13:55:42,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:55:42,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:42,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. 2023-02-10 13:55:42,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d1e6e5a45d298475173074736dd3b162: Waiting for close lock at 1676037342617Running coprocessor pre-close hooks at 1676037342617Disabling compacts and flushes for region at 1676037342617Disabling writes for close at 1676037342617Obtaining lock to block concurrent updates at 1676037342617Preparing flush snapshotting stores in d1e6e5a45d298475173074736dd3b162 at 1676037342617Finished memstore snapshotting testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1676037342618 (+1 ms)Flushing stores of testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. at 1676037342618Flushing d1e6e5a45d298475173074736dd3b162/cf1: creating writer at 1676037342619 (+1 ms)Flushing d1e6e5a45d298475173074736dd3b162/cf1: appending metadata at 1676037342621 (+2 ms)Flushing d1e6e5a45d298475173074736dd3b162/cf1: closing flushed file at 1676037342621Flushing d1e6e5a45d298475173074736dd3b162/cf2: creating writer at 1676037342629 (+8 ms)Flushing d1e6e5a45d298475173074736dd3b162/cf2: appending metadata at 1676037342631 (+2 ms)Flushing d1e6e5a45d298475173074736dd3b162/cf2: closing flushed file at 1676037342631Flushing d1e6e5a45d298475173074736dd3b162/cf1: reopening flushed file at 1676037342645 (+14 ms)Flushing d1e6e5a45d298475173074736dd3b162/cf2: reopening flushed file at 1676037342649 (+4 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for d1e6e5a45d298475173074736dd3b162 in 35ms, sequenceid=82, compaction requested=true at 1676037342652 (+3 ms)Writing region close event to WAL at 1676037342653 (+1 ms)Running coprocessor post-close hooks at 1676037342656 (+3 ms)Closed at 1676037342657 (+1 ms) 2023-02-10 13:55:42,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,660 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED 2023-02-10 13:55:42,660 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037342660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037342660"}]},"ts":"1676037342660"} 2023-02-10 13:55:42,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=113 2023-02-10 13:55:42,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 in 196 msec 2023-02-10 13:55:42,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=112 2023-02-10 13:55:42,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN in 208 msec 2023-02-10 13:55:42,666 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:42,666 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,666 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:42,666 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:42,705 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 3 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,705 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:42,705 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:42,705 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:42,705 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:42,706 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:42,706 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:42,706 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:42,706 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:42,706 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:42,707 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-10 13:55:42,707 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:42,708 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:42,708 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:42,708 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:42,708 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:42,708 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:42,708 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,709 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,710 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,710 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:42,710 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:42,710 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,710 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,710 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,711 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,712 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,712 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,712 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:42,712 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,712 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,713 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,713 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,713 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,713 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:42,714 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,714 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:42,714 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:42,714 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:42,714 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,714 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,714 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,714 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:42,714 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:42,714 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-02-10 13:55:42,715 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:42,715 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:42,715 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037372715 2023-02-10 13:55:42,716 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:42,717 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,719 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:42,719 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:42,719 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:42,720 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037402720 2023-02-10 13:55:42,720 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037462720 2023-02-10 13:55:42,720 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-10 13:55:42,722 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:42,730 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=d1e6e5a45d298475173074736dd3b162, threads=14 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,731 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,732 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,735 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,735 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,735 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,736 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,736 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,851 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,858 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,858 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,859 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,864 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,865 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,871 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,871 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,879 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,884 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,884 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,885 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,887 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,888 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,888 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region d1e6e5a45d298475173074736dd3b162 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:55:42,930 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:42,930 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,930 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:42,930 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:42,970 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 4 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,970 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:42,970 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:42,970 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:42,970 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:42,970 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:42,971 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:42,971 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:42,971 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:42,971 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:42,972 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-10 13:55:42,972 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:42,972 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:42,973 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:42,973 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:42,973 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:42,973 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:42,973 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,973 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:42,974 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:42,975 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,976 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:42,976 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,976 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:42,976 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,976 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:42,977 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,977 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:55:42,977 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,977 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:42,978 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,978 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:42,978 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,978 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,978 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:42,979 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:42,979 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,979 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:42,979 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:42,979 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:42,979 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,980 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,980 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,980 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:42,980 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:42,980 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-02-10 13:55:42,981 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:42,981 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:42,984 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037372984 2023-02-10 13:55:42,985 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:42,985 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:42,988 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:42,988 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:42,988 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:42,988 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037402988 2023-02-10 13:55:42,988 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037462988 2023-02-10 13:55:42,988 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:55:42,997 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=d1e6e5a45d298475173074736dd3b162, threads=14 2023-02-10 13:55:42,997 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,998 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,998 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,998 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,998 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,998 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:42,999 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,001 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,004 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,006 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,007 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,007 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,007 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,010 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,146 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,158 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,158 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,159 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,164 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,165 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,175 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,176 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,177 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,177 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,178 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,179 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,180 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,181 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 for region: d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,181 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region d1e6e5a45d298475173074736dd3b162 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:55:43,208 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:43,213 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:55:43,214 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:55:43,214 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:43,214 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,214 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:43,214 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:43,238 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 5 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:43,239 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:43,239 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:43,240 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:43,241 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-10 13:55:43,241 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:43,241 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:43,241 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:43,241 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,242 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,243 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,244 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,245 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:43,246 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,246 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,246 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:43,246 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:43,246 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-10 13:55:43,247 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:43,247 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:43,248 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037373248 2023-02-10 13:55:43,248 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:43,249 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,251 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:43,252 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:43,252 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:43,252 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037403252 2023-02-10 13:55:43,252 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037463252 2023-02-10 13:55:43,252 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:55:43,254 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:43,256 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:43,257 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:43,257 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:43,502 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 6 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,502 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:43,502 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:43,502 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:43,502 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:43,503 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:43,503 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:43,503 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:43,503 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:43,503 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:43,504 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-10 13:55:43,504 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:43,505 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:43,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:43,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:43,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:43,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:43,505 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,506 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,507 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:43,508 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,509 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:43,510 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:43,510 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:43,510 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,510 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,510 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,510 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:43,510 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:43,510 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-10 13:55:43,511 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:43,511 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:43,511 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037373511 2023-02-10 13:55:43,513 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:43,513 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,515 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:43,515 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:43,515 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:43,515 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037403515 2023-02-10 13:55:43,515 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037463515 2023-02-10 13:55:43,515 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-10 13:55:43,515 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:43,517 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037343517"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1676037343517"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1676037343517"}]},"ts":"1676037343517"} 2023-02-10 13:55:43,517 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037343517"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037343517"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037343517"}]},"ts":"1676037343517"} 2023-02-10 13:55:43,517 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037343517"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037343517"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037343517"}]},"ts":"1676037343517"} 2023-02-10 13:55:43,519 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:43,519 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,519 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:43,519 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:43,579 ERROR [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1527945586-172.31.14.131-1676037238735 (Datanode Uuid 10b8853b-22ce-4aa3-86bd-2e3e9072bae3) service to localhost/127.0.0.1:36875 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-10 13:55:43,765 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 7 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:43,766 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:43,766 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:43,767 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:43,768 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-02-10 13:55:43,768 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:43,768 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:43,768 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:43,769 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:43,769 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:43,769 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:43,769 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,770 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:43,770 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,770 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:43,770 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,770 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:43,770 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,771 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:43,771 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,771 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:43,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:43,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:43,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,772 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:43,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:43,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:43,773 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:43,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,774 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:43,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:43,775 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,776 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,776 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:43,776 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:43,776 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,777 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:43,777 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:43,777 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:43,777 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,777 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,777 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,777 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:43,777 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:43,778 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-02-10 13:55:43,778 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:43,779 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:43,779 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037373779 2023-02-10 13:55:43,781 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,781 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:43,783 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:43,783 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:43,783 INFO [Listener at localhost/43477] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:43,783 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=fe7a6e74cee43286b1157cbd01243f99, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-10 13:55:43,783 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-10 13:55:43,783 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:43,783 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037403783 2023-02-10 13:55:43,784 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037463784 2023-02-10 13:55:43,784 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:55:43,784 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1676037343784"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1676037343784"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1676037343784"}]},"ts":"1676037343784"} 2023-02-10 13:55:43,784 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037343784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037343784"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037343784"}]},"ts":"1676037343784"} 2023-02-10 13:55:43,784 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037343784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037343784"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037343784"}]},"ts":"1676037343784"} 2023-02-10 13:55:43,786 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:43,788 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:43,788 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:43,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:43,788 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 8 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:44,034 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:44,034 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:44,035 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:44,035 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:44,036 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11] 2023-02-10 13:55:44,036 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:44,036 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:44,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:44,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:44,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:44,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:44,037 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,038 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,039 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,040 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,041 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:44,042 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,042 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,042 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,042 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,042 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,042 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-10 13:55:44,042 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:44,043 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:44,043 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037374043 2023-02-10 13:55:44,044 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:44,045 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,047 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:44,047 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:44,047 INFO [Listener at localhost/43477] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:44,047 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=fe7a6e74cee43286b1157cbd01243f99, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-10 13:55:44,047 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-10 13:55:44,048 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:44,048 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037404048 2023-02-10 13:55:44,048 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037464048 2023-02-10 13:55:44,048 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-10 13:55:44,048 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:44,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN}] 2023-02-10 13:55:44,050 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:44,050 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:44,050 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:44,298 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 9 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,298 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:44,298 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:44,298 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:44,298 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:44,298 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:44,298 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:44,299 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:44,299 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:44,299 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:44,300 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12] 2023-02-10 13:55:44,300 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:44,300 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:44,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:44,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:44,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:44,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:44,301 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,302 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,303 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,304 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,304 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:44,304 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,304 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,304 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,304 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,305 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:44,306 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,306 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:44,306 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,306 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:44,306 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,306 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,306 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,306 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,306 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,306 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-10 13:55:44,307 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:44,307 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:44,308 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037374308 2023-02-10 13:55:44,310 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:44,310 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,312 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:44,312 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:44,312 INFO [Listener at localhost/43477] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:44,313 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=fe7a6e74cee43286b1157cbd01243f99, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-10 13:55:44,313 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-10 13:55:44,313 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:44,313 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037404313 2023-02-10 13:55:44,313 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037464313 2023-02-10 13:55:44,313 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:55:44,313 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN}] 2023-02-10 13:55:44,314 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:44,316 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN 2023-02-10 13:55:44,316 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:44,317 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN 2023-02-10 13:55:44,317 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:44,467 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:55:44,468 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,468 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037344468"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037344468"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037344468"}]},"ts":"1676037344468"} 2023-02-10 13:55:44,470 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=116, state=RUNNABLE; OpenRegionProcedure a5e73c1402d9e17b2eb3ac6b51322abc, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:44,472 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=fe7a6e74cee43286b1157cbd01243f99, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,472 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037344472"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037344472"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037344472"}]},"ts":"1676037344472"} 2023-02-10 13:55:44,473 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=118, ppid=115, state=RUNNABLE; OpenRegionProcedure fe7a6e74cee43286b1157cbd01243f99, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:44,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fe7a6e74cee43286b1157cbd01243f99, NAME => 'testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.', STARTKEY => '', ENDKEY => '41'} 2023-02-10 13:55:44,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:44,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:44,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,625 INFO [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,626 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1 2023-02-10 13:55:44,626 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1 2023-02-10 13:55:44,627 INFO [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fe7a6e74cee43286b1157cbd01243f99 columnFamilyName cf1 2023-02-10 13:55:44,640 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce-bottom 2023-02-10 13:55:44,644 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf-bottom 2023-02-10 13:55:44,648 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4-bottom 2023-02-10 13:55:44,652 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857-bottom 2023-02-10 13:55:44,656 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9-bottom 2023-02-10 13:55:44,660 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2-bottom 2023-02-10 13:55:44,665 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613-bottom 2023-02-10 13:55:44,665 INFO [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(310): Store=fe7a6e74cee43286b1157cbd01243f99/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:44,665 INFO [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,666 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2 2023-02-10 13:55:44,666 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2 2023-02-10 13:55:44,667 INFO [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fe7a6e74cee43286b1157cbd01243f99 columnFamilyName cf2 2023-02-10 13:55:44,680 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28-bottom 2023-02-10 13:55:44,685 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c-bottom 2023-02-10 13:55:44,689 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2-bottom 2023-02-10 13:55:44,693 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d-bottom 2023-02-10 13:55:44,696 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5-bottom 2023-02-10 13:55:44,699 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2-bottom 2023-02-10 13:55:44,703 DEBUG [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa-bottom 2023-02-10 13:55:44,703 INFO [StoreOpener-fe7a6e74cee43286b1157cbd01243f99-1] regionserver.HStore(310): Store=fe7a6e74cee43286b1157cbd01243f99/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:44,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:44,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:44,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened fe7a6e74cee43286b1157cbd01243f99; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63968024, jitterRate=-0.04680216312408447}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:44,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for fe7a6e74cee43286b1157cbd01243f99: Running coprocessor pre-open hook at 1676037344624Writing region info on filesystem at 1676037344624Initializing all the Stores at 1676037344625 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037344625Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037344625Cleaning up temporary data from old regions at 1676037344705 (+80 ms)Cleaning up detritus from prior splits at 1676037344706 (+1 ms)Running coprocessor post-open hooks at 1676037344708 (+2 ms)Region opened successfully at 1676037344708 2023-02-10 13:55:44,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99., pid=118, masterSystemTime=1676037344621 2023-02-10 13:55:44,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,711 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:44,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,712 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:44,712 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,712 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): fe7a6e74cee43286b1157cbd01243f99/cf1 is initiating minor compaction (all files) 2023-02-10 13:55:44,712 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,712 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): fe7a6e74cee43286b1157cbd01243f99/cf2 is initiating minor compaction (all files) 2023-02-10 13:55:44,712 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of fe7a6e74cee43286b1157cbd01243f99/cf1 in testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,712 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of fe7a6e74cee43286b1157cbd01243f99/cf2 in testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,713 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/.tmp, totalSize=34.9 K 2023-02-10 13:55:44,713 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/.tmp, totalSize=34.9 K 2023-02-10 13:55:44,713 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037340336 2023-02-10 13:55:44,713 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037340336 2023-02-10 13:55:44,713 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:44,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,714 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037340501 2023-02-10 13:55:44,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a5e73c1402d9e17b2eb3ac6b51322abc, NAME => 'testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.', STARTKEY => '41', ENDKEY => ''} 2023-02-10 13:55:44,714 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=fe7a6e74cee43286b1157cbd01243f99, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:44,714 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037344714"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037344714"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037344714"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037344714"}]},"ts":"1676037344714"} 2023-02-10 13:55:44,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:44,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,715 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037340501 2023-02-10 13:55:44,715 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037340679 2023-02-10 13:55:44,715 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037340679 2023-02-10 13:55:44,715 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037340854 2023-02-10 13:55:44,716 INFO [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,716 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037340854 2023-02-10 13:55:44,716 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037341023 2023-02-10 13:55:44,716 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037341207 2023-02-10 13:55:44,716 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037341023 2023-02-10 13:55:44,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=118, resume processing ppid=115 2023-02-10 13:55:44,717 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037341207 2023-02-10 13:55:44,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure fe7a6e74cee43286b1157cbd01243f99, server=jenkins-hbase4.apache.org,40651,1676037244010 in 242 msec 2023-02-10 13:55:44,717 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037341376 2023-02-10 13:55:44,717 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1 2023-02-10 13:55:44,717 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1 2023-02-10 13:55:44,717 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037341376 2023-02-10 13:55:44,717 INFO [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5e73c1402d9e17b2eb3ac6b51322abc columnFamilyName cf1 2023-02-10 13:55:44,719 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN in 404 msec 2023-02-10 13:55:44,730 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): fe7a6e74cee43286b1157cbd01243f99#cf1#compaction#108 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:44,730 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): fe7a6e74cee43286b1157cbd01243f99#cf2#compaction#109 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:44,733 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce-top 2023-02-10 13:55:44,736 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf-top 2023-02-10 13:55:44,740 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4-top 2023-02-10 13:55:44,742 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/.tmp/cf2/eb3689145d8e482f9b2c24c6793320f2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/eb3689145d8e482f9b2c24c6793320f2 2023-02-10 13:55:44,743 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/.tmp/cf1/4da214095be94995a82ea03468a6de7f as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/4da214095be94995a82ea03468a6de7f 2023-02-10 13:55:44,743 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857-top 2023-02-10 13:55:44,746 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9-top 2023-02-10 13:55:44,747 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fe7a6e74cee43286b1157cbd01243f99/cf2 of fe7a6e74cee43286b1157cbd01243f99 into eb3689145d8e482f9b2c24c6793320f2(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:44,747 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for fe7a6e74cee43286b1157cbd01243f99: 2023-02-10 13:55:44,747 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99., storeName=fe7a6e74cee43286b1157cbd01243f99/cf2, priority=9, startTime=1676037344711; duration=0sec 2023-02-10 13:55:44,747 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,749 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fe7a6e74cee43286b1157cbd01243f99/cf1 of fe7a6e74cee43286b1157cbd01243f99 into 4da214095be94995a82ea03468a6de7f(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:44,749 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for fe7a6e74cee43286b1157cbd01243f99: 2023-02-10 13:55:44,749 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99., storeName=fe7a6e74cee43286b1157cbd01243f99/cf1, priority=9, startTime=1676037344709; duration=0sec 2023-02-10 13:55:44,749 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,750 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2-top 2023-02-10 13:55:44,752 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613-top 2023-02-10 13:55:44,752 INFO [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(310): Store=a5e73c1402d9e17b2eb3ac6b51322abc/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:44,752 INFO [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,753 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2 2023-02-10 13:55:44,753 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2 2023-02-10 13:55:44,753 INFO [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5e73c1402d9e17b2eb3ac6b51322abc columnFamilyName cf2 2023-02-10 13:55:44,763 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28-top 2023-02-10 13:55:44,765 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c-top 2023-02-10 13:55:44,767 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2-top 2023-02-10 13:55:44,770 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d-top 2023-02-10 13:55:44,772 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5-top 2023-02-10 13:55:44,775 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2-top 2023-02-10 13:55:44,778 DEBUG [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa-top 2023-02-10 13:55:44,778 INFO [StoreOpener-a5e73c1402d9e17b2eb3ac6b51322abc-1] regionserver.HStore(310): Store=a5e73c1402d9e17b2eb3ac6b51322abc/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:44,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:44,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a5e73c1402d9e17b2eb3ac6b51322abc; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72877377, jitterRate=0.08595754206180573}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:44,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a5e73c1402d9e17b2eb3ac6b51322abc: Running coprocessor pre-open hook at 1676037344714Writing region info on filesystem at 1676037344714Initializing all the Stores at 1676037344715 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037344715Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037344715Cleaning up temporary data from old regions at 1676037344780 (+65 ms)Cleaning up detritus from prior splits at 1676037344781 (+1 ms)Running coprocessor post-open hooks at 1676037344783 (+2 ms)Region opened successfully at 1676037344783 2023-02-10 13:55:44,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc., pid=117, masterSystemTime=1676037344621 2023-02-10 13:55:44,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,785 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:44,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:55:44,785 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:44,785 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,785 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): a5e73c1402d9e17b2eb3ac6b51322abc/cf1 is initiating minor compaction (all files) 2023-02-10 13:55:44,785 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,785 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): a5e73c1402d9e17b2eb3ac6b51322abc/cf2 is initiating minor compaction (all files) 2023-02-10 13:55:44,785 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of a5e73c1402d9e17b2eb3ac6b51322abc/cf1 in testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,786 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of a5e73c1402d9e17b2eb3ac6b51322abc/cf2 in testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,786 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857-top] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/.tmp, totalSize=34.9 K 2023-02-10 13:55:44,786 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28-top] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/.tmp, totalSize=34.9 K 2023-02-10 13:55:44,786 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037340336 2023-02-10 13:55:44,787 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037340336 2023-02-10 13:55:44,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:44,787 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037340501 2023-02-10 13:55:44,787 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,787 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037340501 2023-02-10 13:55:44,787 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037340679 2023-02-10 13:55:44,787 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037344787"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037344787"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037344787"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037344787"}]},"ts":"1676037344787"} 2023-02-10 13:55:44,788 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037340679 2023-02-10 13:55:44,788 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037340854 2023-02-10 13:55:44,788 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037340854 2023-02-10 13:55:44,788 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1676037341023 2023-02-10 13:55:44,789 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1676037341023 2023-02-10 13:55:44,789 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1676037341207 2023-02-10 13:55:44,789 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1676037341207 2023-02-10 13:55:44,789 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037341376 2023-02-10 13:55:44,790 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037341376 2023-02-10 13:55:44,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=116 2023-02-10 13:55:44,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure a5e73c1402d9e17b2eb3ac6b51322abc, server=jenkins-hbase4.apache.org,40651,1676037244010 in 319 msec 2023-02-10 13:55:44,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=112 2023-02-10 13:55:44,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN in 479 msec 2023-02-10 13:55:44,793 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-10 13:55:44,793 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:44,793 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-10 13:55:44,801 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): a5e73c1402d9e17b2eb3ac6b51322abc#cf2#compaction#111 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:44,801 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5e73c1402d9e17b2eb3ac6b51322abc#cf1#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:44,813 INFO [Listener at localhost/43477] procedure.MasterProcedureTestingUtility(454): Restart 10 exec state=pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,814 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-10 13:55:44,814 INFO [Listener at localhost/43477] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:44,814 INFO [Listener at localhost/43477] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:44,814 ERROR [Listener at localhost/43477] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-10 13:55:44,814 INFO [Listener at localhost/43477] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-10 13:55:44,814 INFO [Listener at localhost/43477] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-10 13:55:44,814 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-10 13:55:44,820 INFO [Listener at localhost/43477] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-10 13:55:44,821 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-10 13:55:44,822 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12, 13] 2023-02-10 13:55:44,822 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-10 13:55:44,822 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-10 13:55:44,822 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=115, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[15, 18, 21] 2023-02-10 13:55:44,822 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=118, ppid=115, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[19, 20] 2023-02-10 13:55:44,822 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=116, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[14, 16, 23] 2023-02-10 13:55:44,823 DEBUG [Listener at localhost/43477] store.ProcedureTree(114): Procedure Procedure(pid=117, ppid=116, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[17, 22] 2023-02-10 13:55:44,823 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-10 13:55:44,823 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-10 13:55:44,823 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-10 13:55:44,823 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-10 13:55:44,824 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,824 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=7a96639de8473c08917073275e5ad055, daughterA=156bb80eea8edfca1cd7027bc93f038b, daughterB=47f7f8f648c73c9a6d503e945e957190 2023-02-10 13:55:44,824 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,824 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-10 13:55:44,824 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,825 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=763a2d94ec8c0a841c2f6b5578c7623c, daughterA=849462bcb81926d6eef14ec7231a1129, daughterB=eda211d0e52e267866f099e27926757d 2023-02-10 13:55:44,825 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,825 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-10 13:55:44,825 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,826 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=9223b687f29223ce102064f5414860f7, daughterA=61b0236e59d564d7f55a2300cc57efa2, daughterB=c09ff3b9bd6b60cc75155fde202ce9ac 2023-02-10 13:55:44,826 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9223b687f29223ce102064f5414860f7, ASSIGN 2023-02-10 13:55:44,826 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,826 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-10 13:55:44,826 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=0cb6d3ddcccf2a26445c39a07e67b5a6, daughterA=68e679423cba5a5772a0f4bdc61f656e, daughterB=61e4dbae3501200f091cac3fbd180757 2023-02-10 13:55:44,827 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/.tmp/cf2/a8da2e7ad38e4033a8d86198554acd64 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/a8da2e7ad38e4033a8d86198554acd64 2023-02-10 13:55:44,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-10 13:55:44,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,827 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=61675fc18532b0447c297a024d6bdc99, daughterA=fa1d3c7bf2e8456cfebc77cba079ea0c, daughterB=b6601c94d8fe2c319f145aba750fb95d 2023-02-10 13:55:44,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-10 13:55:44,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,828 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRollbackForSplitTableRegionWithReplica, region=dd8ff2cf66015ae663ea1192dc3ea15d is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=c2243fb154cf401741526d8ff8577b4c, daughterA=a87fc2ade894946b5fe3f3aacce63525, daughterB=88611e473033c1ce95645cf6a446e0d7 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-10 13:55:44,829 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/.tmp/cf1/cceb6158d46c416e9cce33096b2277b3 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/cceb6158d46c416e9cce33096b2277b3 2023-02-10 13:55:44,829 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure a5e73c1402d9e17b2eb3ac6b51322abc, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(468): Loading pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure fe7a6e74cee43286b1157cbd01243f99, server=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc held the lock before restarting, call acquireLock to restore it. 2023-02-10 13:55:44,831 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,831 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,831 INFO [Listener at localhost/43477] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,831 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,832 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure d1e6e5a45d298475173074736dd3b162, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,832 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,832 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,832 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure a5e73c1402d9e17b2eb3ac6b51322abc, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,832 DEBUG [Listener at localhost/43477] procedure2.Procedure(946): pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure fe7a6e74cee43286b1157cbd01243f99, server=jenkins-hbase4.apache.org,40651,1676037244010 didn't hold the lock before restarting, skip acquiring lock. 2023-02-10 13:55:44,832 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 11 msec 2023-02-10 13:55:44,833 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-10 13:55:44,833 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39591,1676037243910, table=hbase:meta, region=1588230740 2023-02-10 13:55:44,835 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a5e73c1402d9e17b2eb3ac6b51322abc/cf2 of a5e73c1402d9e17b2eb3ac6b51322abc into a8da2e7ad38e4033a8d86198554acd64(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:44,835 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a5e73c1402d9e17b2eb3ac6b51322abc/cf1 of a5e73c1402d9e17b2eb3ac6b51322abc into cceb6158d46c416e9cce33096b2277b3(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:44,835 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for a5e73c1402d9e17b2eb3ac6b51322abc: 2023-02-10 13:55:44,835 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676037374835 2023-02-10 13:55:44,836 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:44,836 DEBUG [Listener at localhost/43477] assignment.AssignmentManager(1519): Joining cluster... 2023-02-10 13:55:44,835 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc., storeName=a5e73c1402d9e17b2eb3ac6b51322abc/cf2, priority=9, startTime=1676037344785; duration=0sec 2023-02-10 13:55:44,835 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for a5e73c1402d9e17b2eb3ac6b51322abc: 2023-02-10 13:55:44,837 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc., storeName=a5e73c1402d9e17b2eb3ac6b51322abc/cf1, priority=9, startTime=1676037344784; duration=0sec 2023-02-10 13:55:44,837 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,837 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:44,839 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=68b3720325c3cd17f2ad6817c681aaf1, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39591,1676037243910, regionLocation=jenkins-hbase4.apache.org,39591,1676037243910, openSeqNum=2 2023-02-10 13:55:44,840 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=d1e6e5a45d298475173074736dd3b162, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=2 2023-02-10 13:55:44,840 INFO [Listener at localhost/43477] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,40651,1676037244010, table=testRecoveryAndDoubleExecution, region=d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:44,840 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=fe7a6e74cee43286b1157cbd01243f99, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=86 2023-02-10 13:55:44,840 DEBUG [Listener at localhost/43477] assignment.RegionStateStore(147): Load hbase:meta entry region=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,40651,1676037244010, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010, openSeqNum=86 2023-02-10 13:55:44,840 INFO [Listener at localhost/43477] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-10 13:55:44,840 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676037404840 2023-02-10 13:55:44,840 INFO [Listener at localhost/43477] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676037464840 2023-02-10 13:55:44,840 INFO [Listener at localhost/43477] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-10 13:55:44,840 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-10 13:55:44,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, state=SUCCESS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=d1e6e5a45d298475173074736dd3b162, daughterA=fe7a6e74cee43286b1157cbd01243f99, daughterB=a5e73c1402d9e17b2eb3ac6b51322abc in 3.2060 sec 2023-02-10 13:55:45,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40651] ipc.CallRunner(144): callId: 858 service: ClientService methodName: Scan size: 154 connection: 172.31.14.131:54866 deadline: 1676037405091, exception=org.apache.hadoop.hbase.NotServingRegionException: testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162. is not online on jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:45,203 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:55:45,209 INFO [Listener at localhost/43477] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-10 13:55:45,210 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:55:45,220 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:45,221 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testRecoveryAndDoubleExecution 2023-02-10 13:55:45,222 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRecoveryAndDoubleExecution 2023-02-10 13:55:45,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-10 13:55:45,225 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037345225"}]},"ts":"1676037345225"} 2023-02-10 13:55:45,226 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLING in hbase:meta 2023-02-10 13:55:45,227 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRecoveryAndDoubleExecution to state=DISABLING 2023-02-10 13:55:45,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, UNASSIGN}, {pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, UNASSIGN}] 2023-02-10 13:55:45,228 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, UNASSIGN 2023-02-10 13:55:45,229 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:45,229 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037345229"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037345229"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037345229"}]},"ts":"1676037345229"} 2023-02-10 13:55:45,230 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=122, ppid=121, state=RUNNABLE; CloseRegionProcedure a5e73c1402d9e17b2eb3ac6b51322abc, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:45,232 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, UNASSIGN 2023-02-10 13:55:45,232 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=fe7a6e74cee43286b1157cbd01243f99, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:45,232 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037345232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037345232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037345232"}]},"ts":"1676037345232"} 2023-02-10 13:55:45,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=123, ppid=120, state=RUNNABLE; CloseRegionProcedure fe7a6e74cee43286b1157cbd01243f99, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:45,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-10 13:55:45,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:45,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing fe7a6e74cee43286b1157cbd01243f99, disabling compactions & flushes 2023-02-10 13:55:45,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:45,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:45,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. after waiting 0 ms 2023-02-10 13:55:45,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:45,382 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857-bottom] to archive 2023-02-10 13:55:45,383 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:45,385 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,386 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,387 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,388 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,389 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,390 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,391 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,392 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28-bottom] to archive 2023-02-10 13:55:45,392 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:45,393 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,394 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,395 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,396 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,396 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,397 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,398 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:55:45,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:45,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99. 2023-02-10 13:55:45,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for fe7a6e74cee43286b1157cbd01243f99: Waiting for close lock at 1676037345382Running coprocessor pre-close hooks at 1676037345382Disabling compacts and flushes for region at 1676037345382Disabling writes for close at 1676037345382Writing region close event to WAL at 1676037345399 (+17 ms)Running coprocessor post-close hooks at 1676037345402 (+3 ms)Closed at 1676037345402 2023-02-10 13:55:45,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:45,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:45,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a5e73c1402d9e17b2eb3ac6b51322abc, disabling compactions & flushes 2023-02-10 13:55:45,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:45,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:45,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. after waiting 0 ms 2023-02-10 13:55:45,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:45,405 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=fe7a6e74cee43286b1157cbd01243f99, regionState=CLOSED 2023-02-10 13:55:45,405 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037345405"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037345405"}]},"ts":"1676037345405"} 2023-02-10 13:55:45,406 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857-top] to archive 2023-02-10 13:55:45,406 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:45,408 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-02-10 13:55:45,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; CloseRegionProcedure fe7a6e74cee43286b1157cbd01243f99, server=jenkins-hbase4.apache.org,40651,1676037244010 in 173 msec 2023-02-10 13:55:45,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=fe7a6e74cee43286b1157cbd01243f99, UNASSIGN in 180 msec 2023-02-10 13:55:45,409 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,410 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,411 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,412 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,413 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,414 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,415 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28-top] to archive 2023-02-10 13:55:45,416 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:45,417 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,418 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,419 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,420 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,421 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,421 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,422 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:55:45,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:45,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc. 2023-02-10 13:55:45,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a5e73c1402d9e17b2eb3ac6b51322abc: Waiting for close lock at 1676037345404Running coprocessor pre-close hooks at 1676037345404Disabling compacts and flushes for region at 1676037345404Disabling writes for close at 1676037345404Writing region close event to WAL at 1676037345423 (+19 ms)Running coprocessor post-close hooks at 1676037345425 (+2 ms)Closed at 1676037345426 (+1 ms) 2023-02-10 13:55:45,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:45,428 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=a5e73c1402d9e17b2eb3ac6b51322abc, regionState=CLOSED 2023-02-10 13:55:45,428 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1676037345428"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037345428"}]},"ts":"1676037345428"} 2023-02-10 13:55:45,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=121 2023-02-10 13:55:45,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=121, state=SUCCESS; CloseRegionProcedure a5e73c1402d9e17b2eb3ac6b51322abc, server=jenkins-hbase4.apache.org,40651,1676037244010 in 199 msec 2023-02-10 13:55:45,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=121, resume processing ppid=119 2023-02-10 13:55:45,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=a5e73c1402d9e17b2eb3ac6b51322abc, UNASSIGN in 202 msec 2023-02-10 13:55:45,432 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037345431"}]},"ts":"1676037345431"} 2023-02-10 13:55:45,432 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLED in hbase:meta 2023-02-10 13:55:45,434 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRecoveryAndDoubleExecution to state=DISABLED 2023-02-10 13:55:45,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; DisableTableProcedure table=testRecoveryAndDoubleExecution in 212 msec 2023-02-10 13:55:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-10 13:55:45,527 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRecoveryAndDoubleExecution, procId: 119 completed 2023-02-10 13:55:45,527 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRecoveryAndDoubleExecution 2023-02-10 13:55:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,530 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,530 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=124, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-02-10 13:55:45,534 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,534 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:45,534 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:45,536 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/recovered.edits] 2023-02-10 13:55:45,536 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/recovered.edits] 2023-02-10 13:55:45,536 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/recovered.edits] 2023-02-10 13:55:45,539 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/4da214095be94995a82ea03468a6de7f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf1/4da214095be94995a82ea03468a6de7f 2023-02-10 13:55:45,539 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/03d29cf5d6b74147a22387e625b3b9ce 2023-02-10 13:55:45,541 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/479381402ff6407ba4e10af6ae2160cf 2023-02-10 13:55:45,542 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/.regioninfo to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/.regioninfo 2023-02-10 13:55:45,542 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/eb3689145d8e482f9b2c24c6793320f2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/cf2/eb3689145d8e482f9b2c24c6793320f2 2023-02-10 13:55:45,542 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/6d0bbef3c7724dd2b73beed0816bcaa4 2023-02-10 13:55:45,543 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/781e33094eeb4a09a7f361994eff3857 2023-02-10 13:55:45,545 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/.regioninfo to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/.regioninfo 2023-02-10 13:55:45,545 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,545 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/80eaed8e110641358db46bf070937cd9 2023-02-10 13:55:45,546 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/9678d2d995fa4e01add3adf639c1a8b2 2023-02-10 13:55:45,546 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,548 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/03d29cf5d6b74147a22387e625b3b9ce.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,548 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf1/b5cfda0e01774a8698fa2182026bd613 2023-02-10 13:55:45,548 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,550 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,550 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/479381402ff6407ba4e10af6ae2160cf.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,551 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/6d0bbef3c7724dd2b73beed0816bcaa4.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,551 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,551 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/0ef95dd82c1c4eb5afe2512e9083bd28 2023-02-10 13:55:45,552 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/781e33094eeb4a09a7f361994eff3857.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,552 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,552 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/1269f987775e47b892c4d726ac112b4c 2023-02-10 13:55:45,553 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/80eaed8e110641358db46bf070937cd9.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,554 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/4ccd28f5ae02471789d62512f3c7aed2 2023-02-10 13:55:45,555 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/9678d2d995fa4e01add3adf639c1a8b2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,555 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/51814fa3b178448a9c8139bccaa35e9d 2023-02-10 13:55:45,556 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,556 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf1/b5cfda0e01774a8698fa2182026bd613.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,556 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5 2023-02-10 13:55:45,560 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d306a3cafc7d40bf8719fe646d6e9ff2 2023-02-10 13:55:45,561 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/cf2/d5219efa2d734ef1883c1c9a0b693bfa 2023-02-10 13:55:45,561 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,561 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/0ef95dd82c1c4eb5afe2512e9083bd28.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,563 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,563 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/1269f987775e47b892c4d726ac112b4c.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,565 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,565 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/recovered.edits/85.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162/recovered.edits/85.seqid 2023-02-10 13:55:45,565 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/4ccd28f5ae02471789d62512f3c7aed2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,566 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,566 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,567 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/51814fa3b178448a9c8139bccaa35e9d.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,568 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,568 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/8e1e05b8db934a0a9ed9e75b2a609cd5.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,569 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,570 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/d306a3cafc7d40bf8719fe646d6e9ff2.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,570 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,571 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/fe7a6e74cee43286b1157cbd01243f99/cf2/d5219efa2d734ef1883c1c9a0b693bfa.d1e6e5a45d298475173074736dd3b162 2023-02-10 13:55:45,573 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/cceb6158d46c416e9cce33096b2277b3 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf1/cceb6158d46c416e9cce33096b2277b3 2023-02-10 13:55:45,574 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99/recovered.edits/90.seqid 2023-02-10 13:55:45,575 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/fe7a6e74cee43286b1157cbd01243f99 2023-02-10 13:55:45,575 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/a8da2e7ad38e4033a8d86198554acd64 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/cf2/a8da2e7ad38e4033a8d86198554acd64 2023-02-10 13:55:45,577 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc/recovered.edits/90.seqid 2023-02-10 13:55:45,577 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testRecoveryAndDoubleExecution/a5e73c1402d9e17b2eb3ac6b51322abc 2023-02-10 13:55:45,578 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-02-10 13:55:45,579 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=124, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,581 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testRecoveryAndDoubleExecution from hbase:meta 2023-02-10 13:55:45,583 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRecoveryAndDoubleExecution' descriptor. 2023-02-10 13:55:45,583 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=124, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,583 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRecoveryAndDoubleExecution' from region states. 2023-02-10 13:55:45,584 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037345583"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:45,584 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037345583"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:45,584 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037345583"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:45,585 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:55:45,585 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d1e6e5a45d298475173074736dd3b162, NAME => 'testRecoveryAndDoubleExecution,,1676037339832.d1e6e5a45d298475173074736dd3b162.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}, {ENCODED => fe7a6e74cee43286b1157cbd01243f99, NAME => 'testRecoveryAndDoubleExecution,,1676037341634.fe7a6e74cee43286b1157cbd01243f99.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => a5e73c1402d9e17b2eb3ac6b51322abc, NAME => 'testRecoveryAndDoubleExecution,41,1676037341634.a5e73c1402d9e17b2eb3ac6b51322abc.', STARTKEY => '41', ENDKEY => ''}] 2023-02-10 13:55:45,585 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRecoveryAndDoubleExecution' as deleted. 2023-02-10 13:55:45,585 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037345585"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:45,586 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRecoveryAndDoubleExecution state from META 2023-02-10 13:55:45,589 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=124, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-10 13:55:45,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, state=SUCCESS; DeleteTableProcedure table=testRecoveryAndDoubleExecution in 62 msec 2023-02-10 13:55:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-02-10 13:55:45,631 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRecoveryAndDoubleExecution, procId: 124 completed 2023-02-10 13:55:45,646 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=375 (was 365) - Thread LEAK? -, OpenFileDescriptor=838 (was 742) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=76 (was 74) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3186 (was 3204) 2023-02-10 13:55:45,660 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=375, OpenFileDescriptor=838, MaxFileDescriptor=60000, SystemLoadAverage=76, ProcessCount=168, AvailableMemoryMB=3186 2023-02-10 13:55:45,660 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:45,662 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:55:45,662 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:55:45,663 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:45,665 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:55:45,665 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:55:45,667 DEBUG [HFileArchiver-16] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:45,667 DEBUG [HFileArchiver-16] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb empty. 2023-02-10 13:55:45,668 DEBUG [HFileArchiver-16] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:45,668 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-02-10 13:55:45,677 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/.tabledesc/.tableinfo.0000000001 2023-02-10 13:55:45,678 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(7671): creating {ENCODED => d88db789cae35cabcb18fcc26dd6defb, NAME => 'testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionEmptyDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:55:45,687 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:45,687 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1603): Closing d88db789cae35cabcb18fcc26dd6defb, disabling compactions & flushes 2023-02-10 13:55:45,687 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:45,687 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:45,687 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. after waiting 0 ms 2023-02-10 13:55:45,687 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:45,687 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:45,687 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1557): Region close journal for d88db789cae35cabcb18fcc26dd6defb: Waiting for close lock at 1676037345687Disabling compacts and flushes for region at 1676037345687Disabling writes for close at 1676037345687Writing region close event to WAL at 1676037345687Closed at 1676037345687 2023-02-10 13:55:45,689 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:55:45,690 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037345689"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037345689"}]},"ts":"1676037345689"} 2023-02-10 13:55:45,691 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:55:45,691 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:55:45,692 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037345691"}]},"ts":"1676037345691"} 2023-02-10 13:55:45,692 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLING in hbase:meta 2023-02-10 13:55:45,695 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:55:45,696 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:55:45,696 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:55:45,696 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:55:45,696 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:55:45,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, ASSIGN}] 2023-02-10 13:55:45,697 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, ASSIGN 2023-02-10 13:55:45,697 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:45,848 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:55:45,849 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=d88db789cae35cabcb18fcc26dd6defb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:45,849 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037345849"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037345849"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037345849"}]},"ts":"1676037345849"} 2023-02-10 13:55:45,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure d88db789cae35cabcb18fcc26dd6defb, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:46,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d88db789cae35cabcb18fcc26dd6defb, NAME => 'testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:55:46,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:46,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:46,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,006 INFO [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,007 DEBUG [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1 2023-02-10 13:55:46,007 DEBUG [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1 2023-02-10 13:55:46,007 INFO [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d88db789cae35cabcb18fcc26dd6defb columnFamilyName cf1 2023-02-10 13:55:46,008 INFO [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] regionserver.HStore(310): Store=d88db789cae35cabcb18fcc26dd6defb/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:46,008 INFO [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,009 DEBUG [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2 2023-02-10 13:55:46,009 DEBUG [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2 2023-02-10 13:55:46,009 INFO [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d88db789cae35cabcb18fcc26dd6defb columnFamilyName cf2 2023-02-10 13:55:46,010 INFO [StoreOpener-d88db789cae35cabcb18fcc26dd6defb-1] regionserver.HStore(310): Store=d88db789cae35cabcb18fcc26dd6defb/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:46,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:46,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:46,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:55:46,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d88db789cae35cabcb18fcc26dd6defb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63223967, jitterRate=-0.05788947641849518}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:46,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d88db789cae35cabcb18fcc26dd6defb: Running coprocessor pre-open hook at 1676037346005Writing region info on filesystem at 1676037346005Initializing all the Stores at 1676037346006 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037346006Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037346006Cleaning up temporary data from old regions at 1676037346011 (+5 ms)Cleaning up detritus from prior splits at 1676037346011Running coprocessor post-open hooks at 1676037346014 (+3 ms)Region opened successfully at 1676037346014 2023-02-10 13:55:46,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb., pid=127, masterSystemTime=1676037346002 2023-02-10 13:55:46,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,017 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=d88db789cae35cabcb18fcc26dd6defb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,017 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037346016"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037346016"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037346016"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037346016"}]},"ts":"1676037346016"} 2023-02-10 13:55:46,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-02-10 13:55:46,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure d88db789cae35cabcb18fcc26dd6defb, server=jenkins-hbase4.apache.org,40651,1676037244010 in 168 msec 2023-02-10 13:55:46,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=126, resume processing ppid=125 2023-02-10 13:55:46,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=125, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, ASSIGN in 323 msec 2023-02-10 13:55:46,021 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:55:46,021 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037346021"}]},"ts":"1676037346021"} 2023-02-10 13:55:46,022 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLED in hbase:meta 2023-02-10 13:55:46,026 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:55:46,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionEmptyDaughter in 363 msec 2023-02-10 13:55:46,170 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:46,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,172 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-10 13:55:46,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-10 13:55:46,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,172 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,173 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,173 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,173 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,173 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,174 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,174 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,175 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:46,176 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. started... 2023-02-10 13:55:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,175 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,177 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,177 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,178 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,185 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/17c849f8812f41e487c0961a43d218fe 2023-02-10 13:55:46,195 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/4040aad615d442a4b644d383a2e2beea 2023-02-10 13:55:46,198 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/17c849f8812f41e487c0961a43d218fe as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe 2023-02-10 13:55:46,201 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:46,202 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/4040aad615d442a4b644d383a2e2beea as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea 2023-02-10 13:55:46,204 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:46,205 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 28ms, sequenceid=6, compaction requested=false 2023-02-10 13:55:46,205 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,206 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d88db789cae35cabcb18fcc26dd6defb: 2023-02-10 13:55:46,206 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,207 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-10 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e23976f[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:46,209 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-10 13:55:46,209 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:46,211 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,211 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,211 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:46,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,211 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,212 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,214 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:55:46,217 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-10 13:55:46,217 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:46,217 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:46,217 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,217 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,220 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,221 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,223 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,223 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,223 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,230 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,230 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:46,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,230 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:46,230 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,331 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:46,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:46,341 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,343 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-10 13:55:46,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-10 13:55:46,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,343 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,344 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,344 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,344 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,344 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,345 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,345 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:46,347 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,347 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,347 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. started... 2023-02-10 13:55:46,347 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,355 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,355 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,355 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,360 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/827fdff6079747a1b9c2a23724b4ee47 2023-02-10 13:55:46,369 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/cda82949d8044a12be38c2155ce07f88 2023-02-10 13:55:46,373 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/827fdff6079747a1b9c2a23724b4ee47 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47 2023-02-10 13:55:46,376 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:46,377 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/cda82949d8044a12be38c2155ce07f88 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88 2023-02-10 13:55:46,380 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:46,381 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 34ms, sequenceid=19, compaction requested=false 2023-02-10 13:55:46,381 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d88db789cae35cabcb18fcc26dd6defb: 2023-02-10 13:55:46,381 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,383 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-10 13:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744925b6[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:46,385 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-10 13:55:46,385 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,386 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,387 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:46,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:46,389 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,389 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,389 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,389 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,389 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,389 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,390 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:55:46,393 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-10 13:55:46,393 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:46,393 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:46,393 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,393 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:46,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,395 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,395 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,395 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,395 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,397 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,397 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,397 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,402 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:46,401 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,401 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,402 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,403 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,403 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,403 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,503 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:46,514 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,515 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,515 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,515 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-10 13:55:46,515 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-10 13:55:46,515 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,515 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,516 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,517 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,516 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,516 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,517 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,517 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,518 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,519 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,519 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,519 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. started... 2023-02-10 13:55:46,519 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:46,520 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,530 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/ca91e35e642b41e2bde544badd3f55d6 2023-02-10 13:55:46,541 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/88456d1200e6455e9b4880c7d58605a5 2023-02-10 13:55:46,544 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/ca91e35e642b41e2bde544badd3f55d6 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6 2023-02-10 13:55:46,547 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:46,548 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/88456d1200e6455e9b4880c7d58605a5 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5 2023-02-10 13:55:46,551 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:46,552 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 33ms, sequenceid=32, compaction requested=true 2023-02-10 13:55:46,552 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d88db789cae35cabcb18fcc26dd6defb: 2023-02-10 13:55:46,552 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,555 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,555 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,555 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,555 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-10 13:55:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5adad285[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:46,557 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-10 13:55:46,557 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,558 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:46,559 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,564 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:55:46,568 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-10 13:55:46,568 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:46,568 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:46,568 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,568 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:46,569 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,569 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,569 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,569 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,569 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,570 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,569 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,570 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,570 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,571 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,572 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,573 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,573 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,576 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,576 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:46,576 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,576 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,677 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:46,688 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,689 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,690 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,690 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-10 13:55:46,690 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-10 13:55:46,690 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,690 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,691 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,691 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,691 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,691 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,692 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,692 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,693 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:46,694 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:46,694 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,694 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. started... 2023-02-10 13:55:46,694 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:46,695 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,702 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/b870b39dfffc44a78eecf061bed2f3c2 2023-02-10 13:55:46,712 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/577d77c17a3e4d8ea082d10a480e0da2 2023-02-10 13:55:46,716 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/b870b39dfffc44a78eecf061bed2f3c2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2 2023-02-10 13:55:46,719 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:46,720 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/577d77c17a3e4d8ea082d10a480e0da2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2 2023-02-10 13:55:46,723 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:46,723 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 29ms, sequenceid=45, compaction requested=true 2023-02-10 13:55:46,723 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d88db789cae35cabcb18fcc26dd6defb: 2023-02-10 13:55:46,724 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,724 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:46,724 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:46,724 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,724 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,724 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,725 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-10 13:55:46,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aeb1ae9[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:46,727 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-10 13:55:46,727 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,728 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,728 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,729 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:46,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,728 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:46,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:46,730 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,730 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,730 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,731 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:46,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:55:46,734 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-10 13:55:46,734 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:46,734 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:46,734 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,734 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,736 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,737 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,737 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,738 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,739 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,739 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,744 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,744 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,744 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,744 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,745 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,845 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:46,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:46,856 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:46,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:46,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,858 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,858 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,858 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-10 13:55:46,858 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-10 13:55:46,858 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,858 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,859 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,859 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,860 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,860 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:46,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:46,862 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,862 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. started... 2023-02-10 13:55:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,862 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:46,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:46,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,862 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,863 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,863 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,863 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,864 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,864 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,864 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,870 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/f64c53e286934c63ade71260ef536523 2023-02-10 13:55:46,882 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/fd586255181a414d96793d6d22b7b8a1 2023-02-10 13:55:46,886 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/f64c53e286934c63ade71260ef536523 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523 2023-02-10 13:55:46,889 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:46,889 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/fd586255181a414d96793d6d22b7b8a1 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1 2023-02-10 13:55:46,892 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:46,893 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 31ms, sequenceid=58, compaction requested=true 2023-02-10 13:55:46,893 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d88db789cae35cabcb18fcc26dd6defb: 2023-02-10 13:55:46,893 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:46,893 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:46,895 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,896 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-10 13:55:46,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78516634[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:46,898 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-10 13:55:46,898 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,899 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:46,899 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,901 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:55:46,905 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-10 13:55:46,905 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:46,905 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:46,905 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,905 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:46,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:46,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,908 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,908 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,908 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,908 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,910 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:46,910 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,911 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,913 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:46,913 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,913 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:46,913 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:46,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:46,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:46,914 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,014 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:47,024 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:47,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:47,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,026 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:47,026 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:47,026 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-10 13:55:47,026 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-10 13:55:47,026 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,026 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,027 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:47,027 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:47,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:47,027 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,027 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:47,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:47,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:47,028 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,028 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:47,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:47,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-10 13:55:47,030 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:47,030 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:47,030 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:47,030 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. started... 2023-02-10 13:55:47,030 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:47,030 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,032 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:47,038 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/6213e802c61b40ea901eba0d0d2391bb 2023-02-10 13:55:47,048 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/486d56327b0647749cd68563a812db35 2023-02-10 13:55:47,051 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/6213e802c61b40ea901eba0d0d2391bb as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb 2023-02-10 13:55:47,055 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:47,055 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/486d56327b0647749cd68563a812db35 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35 2023-02-10 13:55:47,058 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:47,059 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 29ms, sequenceid=71, compaction requested=true 2023-02-10 13:55:47,059 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d88db789cae35cabcb18fcc26dd6defb: 2023-02-10 13:55:47,059 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-10 13:55:47,059 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-10 13:55:47,061 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:47,061 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,061 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-10 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4588981d[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:47,063 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-10 13:55:47,063 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:47,066 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,066 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:47,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,066 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:47,066 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:47,066 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:47,066 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,067 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-10 13:55:47,067 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-10 13:55:47,067 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:47,068 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:47,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-10 13:55:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:47,069 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-10 13:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:47,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-10 13:55:47,072 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-10 13:55:47,072 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:47,072 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:47,072 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,072 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:47,074 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:47,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:47,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,075 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,075 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:47,077 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,077 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,077 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:47,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:47,080 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:47,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:47,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:47,080 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-10 13:55:47,081 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:47,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,081 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,081 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:47,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:47,080 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:47,081 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:47,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:47,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:47,081 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:47,081 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:47,081 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-10 13:55:47,182 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:47,189 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,190 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=d88db789cae35cabcb18fcc26dd6defb, daughterA=81468f2646fa9d3d29c30ebf526e0b3c, daughterB=3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,191 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=d88db789cae35cabcb18fcc26dd6defb, daughterA=81468f2646fa9d3d29c30ebf526e0b3c, daughterB=3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,191 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=d88db789cae35cabcb18fcc26dd6defb, daughterA=81468f2646fa9d3d29c30ebf526e0b3c, daughterB=3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,191 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=d88db789cae35cabcb18fcc26dd6defb, daughterA=81468f2646fa9d3d29c30ebf526e0b3c, daughterB=3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,193 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, UNASSIGN}] 2023-02-10 13:55:47,193 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, UNASSIGN 2023-02-10 13:55:47,194 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=d88db789cae35cabcb18fcc26dd6defb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,194 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037347194"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037347194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037347194"}]},"ts":"1676037347194"} 2023-02-10 13:55:47,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=129, state=RUNNABLE; CloseRegionProcedure d88db789cae35cabcb18fcc26dd6defb, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:47,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d88db789cae35cabcb18fcc26dd6defb, disabling compactions & flushes 2023-02-10 13:55:47,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:47,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:47,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. after waiting 0 ms 2023-02-10 13:55:47,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:47,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d88db789cae35cabcb18fcc26dd6defb 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:55:47,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/d9cadf59d31f421a8d22937a671a0b84 2023-02-10 13:55:47,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/baca8d2e123e4ae2a49af367fd12366b 2023-02-10 13:55:47,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf1/d9cadf59d31f421a8d22937a671a0b84 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84 2023-02-10 13:55:47,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:47,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.tmp/cf2/baca8d2e123e4ae2a49af367fd12366b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b 2023-02-10 13:55:47,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:47,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 32ms, sequenceid=82, compaction requested=true 2023-02-10 13:55:47,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:55:47,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:47,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. 2023-02-10 13:55:47,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d88db789cae35cabcb18fcc26dd6defb: Waiting for close lock at 1676037347347Running coprocessor pre-close hooks at 1676037347347Disabling compacts and flushes for region at 1676037347347Disabling writes for close at 1676037347347Obtaining lock to block concurrent updates at 1676037347347Preparing flush snapshotting stores in d88db789cae35cabcb18fcc26dd6defb at 1676037347347Finished memstore snapshotting testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1676037347347Flushing stores of testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. at 1676037347348 (+1 ms)Flushing d88db789cae35cabcb18fcc26dd6defb/cf1: creating writer at 1676037347350 (+2 ms)Flushing d88db789cae35cabcb18fcc26dd6defb/cf1: appending metadata at 1676037347351 (+1 ms)Flushing d88db789cae35cabcb18fcc26dd6defb/cf1: closing flushed file at 1676037347351Flushing d88db789cae35cabcb18fcc26dd6defb/cf2: creating writer at 1676037347360 (+9 ms)Flushing d88db789cae35cabcb18fcc26dd6defb/cf2: appending metadata at 1676037347362 (+2 ms)Flushing d88db789cae35cabcb18fcc26dd6defb/cf2: closing flushed file at 1676037347362Flushing d88db789cae35cabcb18fcc26dd6defb/cf1: reopening flushed file at 1676037347371 (+9 ms)Flushing d88db789cae35cabcb18fcc26dd6defb/cf2: reopening flushed file at 1676037347375 (+4 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for d88db789cae35cabcb18fcc26dd6defb in 32ms, sequenceid=82, compaction requested=true at 1676037347379 (+4 ms)Writing region close event to WAL at 1676037347381 (+2 ms)Running coprocessor post-close hooks at 1676037347383 (+2 ms)Closed at 1676037347384 (+1 ms) 2023-02-10 13:55:47,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,385 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=d88db789cae35cabcb18fcc26dd6defb, regionState=CLOSED 2023-02-10 13:55:47,385 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037347385"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037347385"}]},"ts":"1676037347385"} 2023-02-10 13:55:47,387 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=129 2023-02-10 13:55:47,387 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=129, state=SUCCESS; CloseRegionProcedure d88db789cae35cabcb18fcc26dd6defb, server=jenkins-hbase4.apache.org,40651,1676037244010 in 191 msec 2023-02-10 13:55:47,389 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=128 2023-02-10 13:55:47,389 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d88db789cae35cabcb18fcc26dd6defb, UNASSIGN in 195 msec 2023-02-10 13:55:47,400 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=128 splitting 14 storefiles, region=d88db789cae35cabcb18fcc26dd6defb, threads=14 2023-02-10 13:55:47,400 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,400 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,414 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,414 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,415 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,415 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,416 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,416 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,416 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,417 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,417 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,417 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,417 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,417 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,419 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea, top=false 2023-02-10 13:55:47,420 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2, top=false 2023-02-10 13:55:47,425 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35, top=false 2023-02-10 13:55:47,426 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6, top=false 2023-02-10 13:55:47,427 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84, top=false 2023-02-10 13:55:47,427 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1, top=false 2023-02-10 13:55:47,429 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88, top=false 2023-02-10 13:55:47,432 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b, top=false 2023-02-10 13:55:47,434 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2, top=false 2023-02-10 13:55:47,434 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523, top=false 2023-02-10 13:55:47,434 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47, top=false 2023-02-10 13:55:47,434 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5, top=false 2023-02-10 13:55:47,436 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb, top=false 2023-02-10 13:55:47,437 DEBUG [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe, top=false 2023-02-10 13:55:47,449 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,449 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,452 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,453 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,453 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,453 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,453 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,456 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,456 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,459 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,462 INFO [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,462 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,462 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,463 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,463 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,464 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,467 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,467 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,469 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,469 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,470 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,470 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,472 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,472 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,472 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/.splits/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47 for child: 81468f2646fa9d3d29c30ebf526e0b3c, parent: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,475 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,475 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,475 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47 for region: d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:47,475 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=128 split storefiles for region d88db789cae35cabcb18fcc26dd6defb Daughter A: 14 storefiles, Daughter B: 0 storefiles. 2023-02-10 13:55:47,506 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:55:47,508 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:55:47,509 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1676037347509"},{"qualifier":"splitA","vlen":69,"tag":[],"timestamp":"1676037347509"},{"qualifier":"splitB","vlen":69,"tag":[],"timestamp":"1676037347509"}]},"ts":"1676037347509"} 2023-02-10 13:55:47,509 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037347509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037347509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037347509"}]},"ts":"1676037347509"} 2023-02-10 13:55:47,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037347509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037347509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037347509"}]},"ts":"1676037347509"} 2023-02-10 13:55:47,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, ASSIGN}, {pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, ASSIGN}] 2023-02-10 13:55:47,513 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, ASSIGN 2023-02-10 13:55:47,513 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:47,514 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, ASSIGN 2023-02-10 13:55:47,514 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:47,664 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:55:47,665 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=81468f2646fa9d3d29c30ebf526e0b3c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,666 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037347665"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037347665"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037347665"}]},"ts":"1676037347665"} 2023-02-10 13:55:47,667 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=133, ppid=131, state=RUNNABLE; OpenRegionProcedure 81468f2646fa9d3d29c30ebf526e0b3c, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:47,669 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=3ebda4753000f25d45d506b180013866, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,669 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037347669"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037347669"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037347669"}]},"ts":"1676037347669"} 2023-02-10 13:55:47,670 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=132, state=RUNNABLE; OpenRegionProcedure 3ebda4753000f25d45d506b180013866, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:47,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:47,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3ebda4753000f25d45d506b180013866, NAME => 'testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.', STARTKEY => '71', ENDKEY => ''} 2023-02-10 13:55:47,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:47,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:47,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,824 INFO [StoreOpener-3ebda4753000f25d45d506b180013866-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,825 DEBUG [StoreOpener-3ebda4753000f25d45d506b180013866-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/cf1 2023-02-10 13:55:47,825 DEBUG [StoreOpener-3ebda4753000f25d45d506b180013866-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/cf1 2023-02-10 13:55:47,826 INFO [StoreOpener-3ebda4753000f25d45d506b180013866-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ebda4753000f25d45d506b180013866 columnFamilyName cf1 2023-02-10 13:55:47,826 INFO [StoreOpener-3ebda4753000f25d45d506b180013866-1] regionserver.HStore(310): Store=3ebda4753000f25d45d506b180013866/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:47,827 INFO [StoreOpener-3ebda4753000f25d45d506b180013866-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,828 DEBUG [StoreOpener-3ebda4753000f25d45d506b180013866-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/cf2 2023-02-10 13:55:47,828 DEBUG [StoreOpener-3ebda4753000f25d45d506b180013866-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/cf2 2023-02-10 13:55:47,828 INFO [StoreOpener-3ebda4753000f25d45d506b180013866-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ebda4753000f25d45d506b180013866 columnFamilyName cf2 2023-02-10 13:55:47,829 INFO [StoreOpener-3ebda4753000f25d45d506b180013866-1] regionserver.HStore(310): Store=3ebda4753000f25d45d506b180013866/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:47,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:47,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:47,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3ebda4753000f25d45d506b180013866; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66371223, jitterRate=-0.010991707444190979}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:47,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3ebda4753000f25d45d506b180013866: Running coprocessor pre-open hook at 1676037347822Writing region info on filesystem at 1676037347822Initializing all the Stores at 1676037347823 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037347823Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037347823Cleaning up temporary data from old regions at 1676037347831 (+8 ms)Cleaning up detritus from prior splits at 1676037347831Running coprocessor post-open hooks at 1676037347833 (+2 ms)Region opened successfully at 1676037347833 2023-02-10 13:55:47,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866., pid=134, masterSystemTime=1676037347818 2023-02-10 13:55:47,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:47,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:47,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:47,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81468f2646fa9d3d29c30ebf526e0b3c, NAME => 'testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.', STARTKEY => '', ENDKEY => '71'} 2023-02-10 13:55:47,836 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=3ebda4753000f25d45d506b180013866, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:47,836 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037347836"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037347836"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037347836"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037347836"}]},"ts":"1676037347836"} 2023-02-10 13:55:47,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:47,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,838 INFO [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,839 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1 2023-02-10 13:55:47,839 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1 2023-02-10 13:55:47,839 INFO [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81468f2646fa9d3d29c30ebf526e0b3c columnFamilyName cf1 2023-02-10 13:55:47,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=132 2023-02-10 13:55:47,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=132, state=SUCCESS; OpenRegionProcedure 3ebda4753000f25d45d506b180013866, server=jenkins-hbase4.apache.org,40651,1676037244010 in 169 msec 2023-02-10 13:55:47,841 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, ASSIGN in 328 msec 2023-02-10 13:55:47,844 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe 2023-02-10 13:55:47,847 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb 2023-02-10 13:55:47,850 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47 2023-02-10 13:55:47,854 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2 2023-02-10 13:55:47,858 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6 2023-02-10 13:55:47,860 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84 2023-02-10 13:55:47,862 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523 2023-02-10 13:55:47,863 INFO [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(310): Store=81468f2646fa9d3d29c30ebf526e0b3c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:47,863 INFO [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,863 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2 2023-02-10 13:55:47,863 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2 2023-02-10 13:55:47,864 INFO [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81468f2646fa9d3d29c30ebf526e0b3c columnFamilyName cf2 2023-02-10 13:55:47,868 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea 2023-02-10 13:55:47,870 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35 2023-02-10 13:55:47,872 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2 2023-02-10 13:55:47,875 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5 2023-02-10 13:55:47,878 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b 2023-02-10 13:55:47,880 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88 2023-02-10 13:55:47,883 DEBUG [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1 2023-02-10 13:55:47,883 INFO [StoreOpener-81468f2646fa9d3d29c30ebf526e0b3c-1] regionserver.HStore(310): Store=81468f2646fa9d3d29c30ebf526e0b3c/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:47,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:47,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:47,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 81468f2646fa9d3d29c30ebf526e0b3c; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59540401, jitterRate=-0.11277888715267181}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:47,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 81468f2646fa9d3d29c30ebf526e0b3c: Running coprocessor pre-open hook at 1676037347837Writing region info on filesystem at 1676037347837Initializing all the Stores at 1676037347837Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037347837Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037347838 (+1 ms)Cleaning up temporary data from old regions at 1676037347885 (+47 ms)Cleaning up detritus from prior splits at 1676037347885Running coprocessor post-open hooks at 1676037347887 (+2 ms)Region opened successfully at 1676037347887 2023-02-10 13:55:47,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c., pid=133, masterSystemTime=1676037347818 2023-02-10 13:55:47,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:47,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:55:47,889 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:47,889 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:47,889 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-10 13:55:47,889 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-10 13:55:47,889 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): 81468f2646fa9d3d29c30ebf526e0b3c/cf1 is initiating minor compaction (all files) 2023-02-10 13:55:47,889 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): 81468f2646fa9d3d29c30ebf526e0b3c/cf2 is initiating minor compaction (all files) 2023-02-10 13:55:47,889 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 81468f2646fa9d3d29c30ebf526e0b3c/cf1 in testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:47,889 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 81468f2646fa9d3d29c30ebf526e0b3c/cf2 in testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:47,889 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/.tmp, totalSize=34.9 K 2023-02-10 13:55:47,889 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/.tmp, totalSize=34.9 K 2023-02-10 13:55:47,890 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037346166 2023-02-10 13:55:47,890 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037346166 2023-02-10 13:55:47,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:47,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:47,890 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037346332 2023-02-10 13:55:47,890 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037346332 2023-02-10 13:55:47,891 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037346504 2023-02-10 13:55:47,891 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=81468f2646fa9d3d29c30ebf526e0b3c, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:47,891 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037346504 2023-02-10 13:55:47,891 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037346678 2023-02-10 13:55:47,891 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037347891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037347891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037347891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037347891"}]},"ts":"1676037347891"} 2023-02-10 13:55:47,891 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037346678 2023-02-10 13:55:47,891 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037346847 2023-02-10 13:55:47,892 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037346847 2023-02-10 13:55:47,892 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037347016 2023-02-10 13:55:47,892 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037347016 2023-02-10 13:55:47,892 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037347183 2023-02-10 13:55:47,893 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037347183 2023-02-10 13:55:47,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=133, resume processing ppid=131 2023-02-10 13:55:47,895 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=133, ppid=131, state=SUCCESS; OpenRegionProcedure 81468f2646fa9d3d29c30ebf526e0b3c, server=jenkins-hbase4.apache.org,40651,1676037244010 in 225 msec 2023-02-10 13:55:47,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=128 2023-02-10 13:55:47,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, ASSIGN in 383 msec 2023-02-10 13:55:47,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=128, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=d88db789cae35cabcb18fcc26dd6defb, daughterA=81468f2646fa9d3d29c30ebf526e0b3c, daughterB=3ebda4753000f25d45d506b180013866 in 707 msec 2023-02-10 13:55:47,916 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): 81468f2646fa9d3d29c30ebf526e0b3c#cf1#compaction#126 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:47,917 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): 81468f2646fa9d3d29c30ebf526e0b3c#cf2#compaction#127 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:47,932 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/.tmp/cf2/c17ff895ace54cca9c51a058c97a2a94 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/c17ff895ace54cca9c51a058c97a2a94 2023-02-10 13:55:47,933 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/.tmp/cf1/227370fd93084dbdbafb46a59b413ceb as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/227370fd93084dbdbafb46a59b413ceb 2023-02-10 13:55:47,937 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 81468f2646fa9d3d29c30ebf526e0b3c/cf2 of 81468f2646fa9d3d29c30ebf526e0b3c into c17ff895ace54cca9c51a058c97a2a94(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:47,937 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 81468f2646fa9d3d29c30ebf526e0b3c/cf1 of 81468f2646fa9d3d29c30ebf526e0b3c into 227370fd93084dbdbafb46a59b413ceb(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:47,937 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 81468f2646fa9d3d29c30ebf526e0b3c: 2023-02-10 13:55:47,937 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 81468f2646fa9d3d29c30ebf526e0b3c: 2023-02-10 13:55:47,937 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c., storeName=81468f2646fa9d3d29c30ebf526e0b3c/cf2, priority=9, startTime=1676037347889; duration=0sec 2023-02-10 13:55:47,937 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c., storeName=81468f2646fa9d3d29c30ebf526e0b3c/cf1, priority=9, startTime=1676037347888; duration=0sec 2023-02-10 13:55:47,937 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:47,938 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:47,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40651] ipc.CallRunner(144): callId: 989 service: ClientService methodName: Scan size: 157 connection: 172.31.14.131:54866 deadline: 1676037407941, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb. is not online on jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:48,051 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '71', inclusive 2023-02-10 13:55:48,055 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:48,057 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,057 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=135, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-02-10 13:55:48,060 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037348060"}]},"ts":"1676037348060"} 2023-02-10 13:55:48,061 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLING in hbase:meta 2023-02-10 13:55:48,063 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionEmptyDaughter to state=DISABLING 2023-02-10 13:55:48,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, UNASSIGN}, {pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, UNASSIGN}] 2023-02-10 13:55:48,064 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, UNASSIGN 2023-02-10 13:55:48,065 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=3ebda4753000f25d45d506b180013866, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:48,065 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037348065"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037348065"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037348065"}]},"ts":"1676037348065"} 2023-02-10 13:55:48,066 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=138, ppid=137, state=RUNNABLE; CloseRegionProcedure 3ebda4753000f25d45d506b180013866, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:48,068 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, UNASSIGN 2023-02-10 13:55:48,068 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=81468f2646fa9d3d29c30ebf526e0b3c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:48,068 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037348068"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037348068"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037348068"}]},"ts":"1676037348068"} 2023-02-10 13:55:48,069 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=139, ppid=136, state=RUNNABLE; CloseRegionProcedure 81468f2646fa9d3d29c30ebf526e0b3c, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:48,158 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:55:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-02-10 13:55:48,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:48,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3ebda4753000f25d45d506b180013866, disabling compactions & flushes 2023-02-10 13:55:48,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:48,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:48,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. after waiting 0 ms 2023-02-10 13:55:48,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:48,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=85 2023-02-10 13:55:48,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:48,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866. 2023-02-10 13:55:48,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3ebda4753000f25d45d506b180013866: Waiting for close lock at 1676037348219Running coprocessor pre-close hooks at 1676037348219Disabling compacts and flushes for region at 1676037348219Disabling writes for close at 1676037348220 (+1 ms)Writing region close event to WAL at 1676037348220Running coprocessor post-close hooks at 1676037348223 (+3 ms)Closed at 1676037348224 (+1 ms) 2023-02-10 13:55:48,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3ebda4753000f25d45d506b180013866 2023-02-10 13:55:48,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:48,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 81468f2646fa9d3d29c30ebf526e0b3c, disabling compactions & flushes 2023-02-10 13:55:48,226 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:48,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:48,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. after waiting 0 ms 2023-02-10 13:55:48,226 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:48,226 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=3ebda4753000f25d45d506b180013866, regionState=CLOSED 2023-02-10 13:55:48,226 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037348226"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037348226"}]},"ts":"1676037348226"} 2023-02-10 13:55:48,226 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84] to archive 2023-02-10 13:55:48,227 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:48,229 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-17c849f8812f41e487c0961a43d218fe 2023-02-10 13:55:48,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=137 2023-02-10 13:55:48,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=137, state=SUCCESS; CloseRegionProcedure 3ebda4753000f25d45d506b180013866, server=jenkins-hbase4.apache.org,40651,1676037244010 in 161 msec 2023-02-10 13:55:48,230 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-827fdff6079747a1b9c2a23724b4ee47 2023-02-10 13:55:48,230 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=3ebda4753000f25d45d506b180013866, UNASSIGN in 166 msec 2023-02-10 13:55:48,231 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-ca91e35e642b41e2bde544badd3f55d6 2023-02-10 13:55:48,233 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-b870b39dfffc44a78eecf061bed2f3c2 2023-02-10 13:55:48,234 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-f64c53e286934c63ade71260ef536523 2023-02-10 13:55:48,235 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-6213e802c61b40ea901eba0d0d2391bb 2023-02-10 13:55:48,236 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-d9cadf59d31f421a8d22937a671a0b84 2023-02-10 13:55:48,237 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b] to archive 2023-02-10 13:55:48,238 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:48,239 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-4040aad615d442a4b644d383a2e2beea 2023-02-10 13:55:48,240 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-cda82949d8044a12be38c2155ce07f88 2023-02-10 13:55:48,241 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-88456d1200e6455e9b4880c7d58605a5 2023-02-10 13:55:48,242 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-577d77c17a3e4d8ea082d10a480e0da2 2023-02-10 13:55:48,242 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-fd586255181a414d96793d6d22b7b8a1 2023-02-10 13:55:48,243 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-486d56327b0647749cd68563a812db35 2023-02-10 13:55:48,244 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/testSplitTableRegionEmptyDaughter=d88db789cae35cabcb18fcc26dd6defb-baca8d2e123e4ae2a49af367fd12366b 2023-02-10 13:55:48,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:55:48,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:48,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c. 2023-02-10 13:55:48,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 81468f2646fa9d3d29c30ebf526e0b3c: Waiting for close lock at 1676037348226Running coprocessor pre-close hooks at 1676037348226Disabling compacts and flushes for region at 1676037348226Disabling writes for close at 1676037348226Writing region close event to WAL at 1676037348245 (+19 ms)Running coprocessor post-close hooks at 1676037348247 (+2 ms)Closed at 1676037348248 (+1 ms) 2023-02-10 13:55:48,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:48,250 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=81468f2646fa9d3d29c30ebf526e0b3c, regionState=CLOSED 2023-02-10 13:55:48,250 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676037348250"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037348250"}]},"ts":"1676037348250"} 2023-02-10 13:55:48,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=139, resume processing ppid=136 2023-02-10 13:55:48,253 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=136, state=SUCCESS; CloseRegionProcedure 81468f2646fa9d3d29c30ebf526e0b3c, server=jenkins-hbase4.apache.org,40651,1676037244010 in 182 msec 2023-02-10 13:55:48,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=135 2023-02-10 13:55:48,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=81468f2646fa9d3d29c30ebf526e0b3c, UNASSIGN in 190 msec 2023-02-10 13:55:48,256 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037348255"}]},"ts":"1676037348255"} 2023-02-10 13:55:48,257 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLED in hbase:meta 2023-02-10 13:55:48,258 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionEmptyDaughter to state=DISABLED 2023-02-10 13:55:48,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=135, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionEmptyDaughter in 202 msec 2023-02-10 13:55:48,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-02-10 13:55:48,362 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 135 completed 2023-02-10 13:55:48,363 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,365 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,366 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=140, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-02-10 13:55:48,369 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:48,369 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:48,369 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866 2023-02-10 13:55:48,371 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/recovered.edits] 2023-02-10 13:55:48,371 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/recovered.edits] 2023-02-10 13:55:48,371 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/recovered.edits] 2023-02-10 13:55:48,374 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/17c849f8812f41e487c0961a43d218fe 2023-02-10 13:55:48,374 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/227370fd93084dbdbafb46a59b413ceb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf1/227370fd93084dbdbafb46a59b413ceb 2023-02-10 13:55:48,375 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/6213e802c61b40ea901eba0d0d2391bb 2023-02-10 13:55:48,376 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/recovered.edits/88.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866/recovered.edits/88.seqid 2023-02-10 13:55:48,376 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/c17ff895ace54cca9c51a058c97a2a94 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/cf2/c17ff895ace54cca9c51a058c97a2a94 2023-02-10 13:55:48,376 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/3ebda4753000f25d45d506b180013866 2023-02-10 13:55:48,376 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/827fdff6079747a1b9c2a23724b4ee47 2023-02-10 13:55:48,377 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/b870b39dfffc44a78eecf061bed2f3c2 2023-02-10 13:55:48,378 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c/recovered.edits/90.seqid 2023-02-10 13:55:48,378 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/ca91e35e642b41e2bde544badd3f55d6 2023-02-10 13:55:48,379 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/81468f2646fa9d3d29c30ebf526e0b3c 2023-02-10 13:55:48,380 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/d9cadf59d31f421a8d22937a671a0b84 2023-02-10 13:55:48,381 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf1/f64c53e286934c63ade71260ef536523 2023-02-10 13:55:48,383 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/4040aad615d442a4b644d383a2e2beea 2023-02-10 13:55:48,384 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/486d56327b0647749cd68563a812db35 2023-02-10 13:55:48,385 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/577d77c17a3e4d8ea082d10a480e0da2 2023-02-10 13:55:48,386 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/88456d1200e6455e9b4880c7d58605a5 2023-02-10 13:55:48,386 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/baca8d2e123e4ae2a49af367fd12366b 2023-02-10 13:55:48,387 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/cda82949d8044a12be38c2155ce07f88 2023-02-10 13:55:48,388 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/cf2/fd586255181a414d96793d6d22b7b8a1 2023-02-10 13:55:48,390 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/recovered.edits/85.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb/recovered.edits/85.seqid 2023-02-10 13:55:48,390 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegionEmptyDaughter/d88db789cae35cabcb18fcc26dd6defb 2023-02-10 13:55:48,390 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-02-10 13:55:48,392 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=140, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,393 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionEmptyDaughter from hbase:meta 2023-02-10 13:55:48,395 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionEmptyDaughter' descriptor. 2023-02-10 13:55:48,395 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=140, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,395 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionEmptyDaughter' from region states. 2023-02-10 13:55:48,396 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037348395"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:48,396 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037348395"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:48,396 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037348395"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:48,397 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:55:48,397 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d88db789cae35cabcb18fcc26dd6defb, NAME => 'testSplitTableRegionEmptyDaughter,,1676037345663.d88db789cae35cabcb18fcc26dd6defb.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 81468f2646fa9d3d29c30ebf526e0b3c, NAME => 'testSplitTableRegionEmptyDaughter,,1676037347189.81468f2646fa9d3d29c30ebf526e0b3c.', STARTKEY => '', ENDKEY => '71'}, {ENCODED => 3ebda4753000f25d45d506b180013866, NAME => 'testSplitTableRegionEmptyDaughter,71,1676037347189.3ebda4753000f25d45d506b180013866.', STARTKEY => '71', ENDKEY => ''}] 2023-02-10 13:55:48,397 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionEmptyDaughter' as deleted. 2023-02-10 13:55:48,397 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037348397"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:48,398 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionEmptyDaughter state from META 2023-02-10 13:55:48,400 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=140, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-10 13:55:48,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter in 37 msec 2023-02-10 13:55:48,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-02-10 13:55:48,467 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 140 completed 2023-02-10 13:55:48,481 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=373 (was 375), OpenFileDescriptor=834 (was 838), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=70 (was 76), ProcessCount=168 (was 168), AvailableMemoryMB=3138 (was 3186) 2023-02-10 13:55:48,493 INFO [Listener at localhost/43477] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=373, OpenFileDescriptor=834, MaxFileDescriptor=60000, SystemLoadAverage=70, ProcessCount=168, AvailableMemoryMB=3137 2023-02-10 13:55:48,493 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:48,496 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-10 13:55:48,496 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-02-10 13:55:48,497 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegion 2023-02-10 13:55:48,499 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_PRE_OPERATION 2023-02-10 13:55:48,499 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-10 13:55:48,500 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,501 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d empty. 2023-02-10 13:55:48,501 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,501 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-02-10 13:55:48,511 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/.tabledesc/.tableinfo.0000000001 2023-02-10 13:55:48,512 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(7671): creating {ENCODED => 13a42a75712db718789d408f760bc81d, NAME => 'testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegion', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp 2023-02-10 13:55:48,518 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:48,518 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1603): Closing 13a42a75712db718789d408f760bc81d, disabling compactions & flushes 2023-02-10 13:55:48,518 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,518 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,518 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. after waiting 0 ms 2023-02-10 13:55:48,518 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,519 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,519 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1557): Region close journal for 13a42a75712db718789d408f760bc81d: Waiting for close lock at 1676037348518Disabling compacts and flushes for region at 1676037348518Disabling writes for close at 1676037348518Writing region close event to WAL at 1676037348519 (+1 ms)Closed at 1676037348519 2023-02-10 13:55:48,520 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ADD_TO_META 2023-02-10 13:55:48,521 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037348521"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037348521"}]},"ts":"1676037348521"} 2023-02-10 13:55:48,522 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-10 13:55:48,523 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-10 13:55:48,523 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037348523"}]},"ts":"1676037348523"} 2023-02-10 13:55:48,524 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLING in hbase:meta 2023-02-10 13:55:48,526 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-10 13:55:48,527 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-10 13:55:48,527 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-10 13:55:48,527 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-10 13:55:48,527 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-10 13:55:48,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, ASSIGN}] 2023-02-10 13:55:48,529 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, ASSIGN 2023-02-10 13:55:48,529 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:48,581 ERROR [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1527945586-172.31.14.131-1676037238735 (Datanode Uuid 10b8853b-22ce-4aa3-86bd-2e3e9072bae3) service to localhost/127.0.0.1:36875 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-10 13:55:48,680 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-10 13:55:48,681 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=13a42a75712db718789d408f760bc81d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:48,681 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037348681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037348681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037348681"}]},"ts":"1676037348681"} 2023-02-10 13:55:48,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=142, state=RUNNABLE; OpenRegionProcedure 13a42a75712db718789d408f760bc81d, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:48,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 13a42a75712db718789d408f760bc81d, NAME => 'testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.', STARTKEY => '', ENDKEY => ''} 2023-02-10 13:55:48,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:48,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:48,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,838 INFO [StoreOpener-13a42a75712db718789d408f760bc81d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,839 DEBUG [StoreOpener-13a42a75712db718789d408f760bc81d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1 2023-02-10 13:55:48,840 DEBUG [StoreOpener-13a42a75712db718789d408f760bc81d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1 2023-02-10 13:55:48,840 INFO [StoreOpener-13a42a75712db718789d408f760bc81d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13a42a75712db718789d408f760bc81d columnFamilyName cf1 2023-02-10 13:55:48,840 INFO [StoreOpener-13a42a75712db718789d408f760bc81d-1] regionserver.HStore(310): Store=13a42a75712db718789d408f760bc81d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:48,840 INFO [StoreOpener-13a42a75712db718789d408f760bc81d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,841 DEBUG [StoreOpener-13a42a75712db718789d408f760bc81d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2 2023-02-10 13:55:48,841 DEBUG [StoreOpener-13a42a75712db718789d408f760bc81d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2 2023-02-10 13:55:48,842 INFO [StoreOpener-13a42a75712db718789d408f760bc81d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 13a42a75712db718789d408f760bc81d columnFamilyName cf2 2023-02-10 13:55:48,842 INFO [StoreOpener-13a42a75712db718789d408f760bc81d-1] regionserver.HStore(310): Store=13a42a75712db718789d408f760bc81d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:48,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:48,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:48,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-10 13:55:48,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 13a42a75712db718789d408f760bc81d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67755854, jitterRate=0.009640902280807495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:48,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 13a42a75712db718789d408f760bc81d: Running coprocessor pre-open hook at 1676037348837Writing region info on filesystem at 1676037348837Initializing all the Stores at 1676037348838 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037348838Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037348838Cleaning up temporary data from old regions at 1676037348843 (+5 ms)Cleaning up detritus from prior splits at 1676037348844 (+1 ms)Running coprocessor post-open hooks at 1676037348847 (+3 ms)Region opened successfully at 1676037348847 2023-02-10 13:55:48,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d., pid=143, masterSystemTime=1676037348834 2023-02-10 13:55:48,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:48,849 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=13a42a75712db718789d408f760bc81d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:48,849 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037348849"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037348849"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037348849"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037348849"}]},"ts":"1676037348849"} 2023-02-10 13:55:48,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=142 2023-02-10 13:55:48,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=142, state=SUCCESS; OpenRegionProcedure 13a42a75712db718789d408f760bc81d, server=jenkins-hbase4.apache.org,40651,1676037244010 in 168 msec 2023-02-10 13:55:48,852 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=141 2023-02-10 13:55:48,852 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=141, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, ASSIGN in 323 msec 2023-02-10 13:55:48,853 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-10 13:55:48,853 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037348853"}]},"ts":"1676037348853"} 2023-02-10 13:55:48,853 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLED in hbase:meta 2023-02-10 13:55:48,855 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_POST_OPERATION 2023-02-10 13:55:48,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, state=SUCCESS; CreateTableProcedure table=testSplitTableRegion in 358 msec 2023-02-10 13:55:49,003 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-10 13:55:49,005 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-10 13:55:49,005 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,005 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-10 13:55:49,005 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-10 13:55:49,005 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,005 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,007 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,007 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,008 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,008 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,009 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:49,010 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:49,010 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,010 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. started... 2023-02-10 13:55:49,011 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-10 13:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,011 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,012 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,020 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/7697412c107c46349bfb4ee5f8b98188 2023-02-10 13:55:49,032 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/90a65bbfb21e4c4183fe699570c6ed16 2023-02-10 13:55:49,035 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/7697412c107c46349bfb4ee5f8b98188 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188 2023-02-10 13:55:49,038 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:49,039 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/90a65bbfb21e4c4183fe699570c6ed16 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16 2023-02-10 13:55:49,042 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:49,042 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 31ms, sequenceid=6, compaction requested=false 2023-02-10 13:55:49,043 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegion' 2023-02-10 13:55:49,043 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 13a42a75712db718789d408f760bc81d: 2023-02-10 13:55:49,043 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,043 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,045 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,045 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,045 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,045 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-10 13:55:49,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@494a34d3[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:49,047 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-10 13:55:49,047 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,049 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:49,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:49,050 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,050 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,051 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:49,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-10 13:55:49,059 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-10 13:55:49,059 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:49,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:49,060 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-10 13:55:49,060 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:49,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,062 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,061 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,062 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,062 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,063 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,064 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,064 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,065 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,067 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-10 13:55:49,068 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,067 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,067 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,068 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,069 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,168 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:49,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:49,178 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:49,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:49,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-10 13:55:49,180 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-10 13:55:49,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-10 13:55:49,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-10 13:55:49,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,180 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,182 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,182 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,182 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,182 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,184 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:49,185 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,185 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. started... 2023-02-10 13:55:49,185 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,196 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/d229eaf9fdba499cb622e3f24c864d73 2023-02-10 13:55:49,209 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/db86f397877940529c57ab9879b3020a 2023-02-10 13:55:49,213 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/d229eaf9fdba499cb622e3f24c864d73 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73 2023-02-10 13:55:49,217 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:49,218 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/db86f397877940529c57ab9879b3020a as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a 2023-02-10 13:55:49,222 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a, entries=10, sequenceid=19, filesize=5.0 K 2023-02-10 13:55:49,222 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 37ms, sequenceid=19, compaction requested=false 2023-02-10 13:55:49,223 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 13a42a75712db718789d408f760bc81d: 2023-02-10 13:55:49,223 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,223 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,226 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,226 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,226 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,226 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-10 13:55:49,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b745341[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:49,229 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-10 13:55:49,229 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,231 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,231 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,232 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:49,233 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,233 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,233 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,233 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,234 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,234 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,234 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,236 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,236 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,236 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:49,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-10 13:55:49,239 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-10 13:55:49,239 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:49,239 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:49,239 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-10 13:55:49,239 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,241 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,242 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,244 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,244 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,244 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,247 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-10 13:55:49,247 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:49,247 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,247 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,248 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,348 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:49,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:49,357 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:49,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:49,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-10 13:55:49,359 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-10 13:55:49,359 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,359 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-10 13:55:49,359 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-10 13:55:49,359 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,359 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,360 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,360 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,360 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,360 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,361 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,361 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,362 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,363 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. started... 2023-02-10 13:55:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,363 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,373 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/feb699dbd8d143d98ffeaa8039c36119 2023-02-10 13:55:49,382 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/ef7e77931b35455e8656fa73e940c023 2023-02-10 13:55:49,385 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/feb699dbd8d143d98ffeaa8039c36119 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119 2023-02-10 13:55:49,388 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:49,389 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/ef7e77931b35455e8656fa73e940c023 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023 2023-02-10 13:55:49,392 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023, entries=10, sequenceid=32, filesize=5.0 K 2023-02-10 13:55:49,392 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 29ms, sequenceid=32, compaction requested=true 2023-02-10 13:55:49,392 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 13a42a75712db718789d408f760bc81d: 2023-02-10 13:55:49,392 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,393 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,393 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,394 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-10 13:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e7fcdd[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:49,396 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-10 13:55:49,396 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:49,397 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,397 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,399 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,400 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,401 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:49,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-10 13:55:49,403 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-10 13:55:49,403 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:49,403 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:49,403 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-10 13:55:49,403 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:49,404 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,404 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,404 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,404 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,404 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,404 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,405 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,405 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,405 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,406 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,407 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,407 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,408 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,416 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,416 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-10 13:55:49,416 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,416 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,417 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,517 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:49,527 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-10 13:55:49,529 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-10 13:55:49,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-10 13:55:49,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-10 13:55:49,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,529 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,530 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,530 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,530 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,530 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,533 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,532 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,533 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. started... 2023-02-10 13:55:49,533 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,534 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,541 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/ae3d6bedbeea4f578c5493040838c699 2023-02-10 13:55:49,555 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/06a28eca1f2e43db8015268fd700ec1f 2023-02-10 13:55:49,559 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/ae3d6bedbeea4f578c5493040838c699 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699 2023-02-10 13:55:49,562 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:49,562 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/06a28eca1f2e43db8015268fd700ec1f as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f 2023-02-10 13:55:49,565 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f, entries=10, sequenceid=45, filesize=5.0 K 2023-02-10 13:55:49,566 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 33ms, sequenceid=45, compaction requested=true 2023-02-10 13:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 13a42a75712db718789d408f760bc81d: 2023-02-10 13:55:49,566 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,566 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,568 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-10 13:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@392ed3ac[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:49,570 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-10 13:55:49,570 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,571 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,571 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,572 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:49,571 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,572 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,574 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-10 13:55:49,578 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-10 13:55:49,578 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:49,578 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:49,578 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-10 13:55:49,578 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,581 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,581 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,582 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,583 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,585 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,585 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,585 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,591 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,591 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-10 13:55:49,591 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,591 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:49,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,592 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,692 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:49,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:49,703 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:49,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:49,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-10 13:55:49,704 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-10 13:55:49,704 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,704 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-10 13:55:49,704 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-10 13:55:49,704 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,704 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,706 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,706 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,706 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,706 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,706 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,706 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,707 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,708 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,708 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. started... 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,708 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,716 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/fb218491ada9461e82e255c50641dded 2023-02-10 13:55:49,725 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/d837d995b5b242c98d15ebd66d23e1f0 2023-02-10 13:55:49,728 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/fb218491ada9461e82e255c50641dded as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded 2023-02-10 13:55:49,730 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:49,731 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/d837d995b5b242c98d15ebd66d23e1f0 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0 2023-02-10 13:55:49,733 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0, entries=10, sequenceid=58, filesize=5.0 K 2023-02-10 13:55:49,734 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 26ms, sequenceid=58, compaction requested=true 2023-02-10 13:55:49,734 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 13a42a75712db718789d408f760bc81d: 2023-02-10 13:55:49,734 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,734 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:49,734 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:49,734 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,734 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,734 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,737 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,737 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,737 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,737 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-10 13:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5b2905[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:49,739 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-10 13:55:49,739 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,740 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:49,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,742 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,743 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-10 13:55:49,745 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-10 13:55:49,745 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:49,745 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:49,745 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-10 13:55:49,745 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:49,748 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,748 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,748 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,748 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,749 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,748 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,749 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,748 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,749 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,750 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,757 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,757 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,758 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,760 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,761 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-10 13:55:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:49,761 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,762 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,762 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,762 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,762 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,762 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,762 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,763 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,763 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,862 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:49,873 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-02-10 13:55:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-10 13:55:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-10 13:55:49,874 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-10 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-10 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-10 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,875 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,875 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,875 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,875 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,876 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,876 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-10 13:55:49,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,878 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,46307,1676037243975' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-10 13:55:49,878 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,878 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-10 13:55:49,878 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-10 13:55:49,878 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,878 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,39591,1676037243910' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,878 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. started... 2023-02-10 13:55:49,878 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-10 13:55:49,879 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,887 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/04ed82e45a3d4d208585210b600f4c52 2023-02-10 13:55:49,896 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/2d3d5e3d01d748f6925875a267062211 2023-02-10 13:55:49,899 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/04ed82e45a3d4d208585210b600f4c52 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52 2023-02-10 13:55:49,901 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:49,902 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/2d3d5e3d01d748f6925875a267062211 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211 2023-02-10 13:55:49,904 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211, entries=10, sequenceid=71, filesize=5.0 K 2023-02-10 13:55:49,905 INFO [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 27ms, sequenceid=71, compaction requested=true 2023-02-10 13:55:49,905 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 13a42a75712db718789d408f760bc81d: 2023-02-10 13:55:49,905 DEBUG [rs(jenkins-hbase4.apache.org,40651,1676037244010)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-10 13:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-10 13:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-10 13:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-10 13:55:49,907 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-10 13:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40651,1676037244010' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-10 13:55:49,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a720f1[Count = 0] remaining members to acquire global barrier 2023-02-10 13:55:49,909 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-10 13:55:49,909 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,910 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-10 13:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,39591,1676037243910' in zk 2023-02-10 13:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,46307,1676037243975' in zk 2023-02-10 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,40651,1676037244010' in zk 2023-02-10 13:55:49,915 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,915 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,915 DEBUG [member: 'jenkins-hbase4.apache.org,39591,1676037243910' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,916 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-10 13:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,40651,1676037244010' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,916 DEBUG [member: 'jenkins-hbase4.apache.org,46307,1676037243975' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-10 13:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,40651,1676037244010': 2023-02-10 13:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40651,1676037244010' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-10 13:55:49,918 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-10 13:55:49,919 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-10 13:55:49,919 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-10 13:55:49,919 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-10 13:55:49,919 INFO [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-10 13:55:49,921 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,920 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-10 13:55:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,921 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-10 13:55:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,921 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-10 13:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,923 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-10 13:55:49,923 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-10 13:55:49,923 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-10 13:55:49,926 DEBUG [(jenkins-hbase4.apache.org,38205,1676037242772)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-10 13:55:49,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-10 13:55:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-10 13:55:49,926 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-10 13:55:49,927 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-10 13:55:49,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,926 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-10 13:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-10 13:55:49,927 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-10 13:55:50,027 DEBUG [Listener at localhost/43477] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-10 13:55:50,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-10 13:55:50,035 DEBUG [Listener at localhost/43477] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:50,035 DEBUG [Listener at localhost/43477] procedure2.ProcedureExecutor(1029): Stored pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=13a42a75712db718789d408f760bc81d, daughterA=f3824dbfbd417a431cb43dca26b1a788, daughterB=c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:50,036 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=13a42a75712db718789d408f760bc81d, daughterA=f3824dbfbd417a431cb43dca26b1a788, daughterB=c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:50,036 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=13a42a75712db718789d408f760bc81d, daughterA=f3824dbfbd417a431cb43dca26b1a788, daughterB=c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:50,036 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=13a42a75712db718789d408f760bc81d, daughterA=f3824dbfbd417a431cb43dca26b1a788, daughterB=c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:50,038 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, UNASSIGN}] 2023-02-10 13:55:50,039 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, UNASSIGN 2023-02-10 13:55:50,039 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=13a42a75712db718789d408f760bc81d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:50,039 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037350039"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037350039"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037350039"}]},"ts":"1676037350039"} 2023-02-10 13:55:50,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=146, ppid=145, state=RUNNABLE; CloseRegionProcedure 13a42a75712db718789d408f760bc81d, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:50,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 13a42a75712db718789d408f760bc81d, disabling compactions & flushes 2023-02-10 13:55:50,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:50,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:50,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. after waiting 0 ms 2023-02-10 13:55:50,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:50,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 13a42a75712db718789d408f760bc81d 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-10 13:55:50,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/eda996ff76dd4781a7df432fba7c3c0b 2023-02-10 13:55:50,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb 2023-02-10 13:55:50,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf1/eda996ff76dd4781a7df432fba7c3c0b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b 2023-02-10 13:55:50,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:50,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/.tmp/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb 2023-02-10 13:55:50,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb, entries=8, sequenceid=82, filesize=5.0 K 2023-02-10 13:55:50,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 32ms, sequenceid=82, compaction requested=true 2023-02-10 13:55:50,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-10 13:55:50,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:50,228 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. 2023-02-10 13:55:50,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 13a42a75712db718789d408f760bc81d: Waiting for close lock at 1676037350192Running coprocessor pre-close hooks at 1676037350192Disabling compacts and flushes for region at 1676037350192Disabling writes for close at 1676037350192Obtaining lock to block concurrent updates at 1676037350192Preparing flush snapshotting stores in 13a42a75712db718789d408f760bc81d at 1676037350192Finished memstore snapshotting testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1676037350193 (+1 ms)Flushing stores of testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. at 1676037350193Flushing 13a42a75712db718789d408f760bc81d/cf1: creating writer at 1676037350194 (+1 ms)Flushing 13a42a75712db718789d408f760bc81d/cf1: appending metadata at 1676037350196 (+2 ms)Flushing 13a42a75712db718789d408f760bc81d/cf1: closing flushed file at 1676037350196Flushing 13a42a75712db718789d408f760bc81d/cf2: creating writer at 1676037350204 (+8 ms)Flushing 13a42a75712db718789d408f760bc81d/cf2: appending metadata at 1676037350206 (+2 ms)Flushing 13a42a75712db718789d408f760bc81d/cf2: closing flushed file at 1676037350206Flushing 13a42a75712db718789d408f760bc81d/cf1: reopening flushed file at 1676037350217 (+11 ms)Flushing 13a42a75712db718789d408f760bc81d/cf2: reopening flushed file at 1676037350220 (+3 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 13a42a75712db718789d408f760bc81d in 32ms, sequenceid=82, compaction requested=true at 1676037350224 (+4 ms)Writing region close event to WAL at 1676037350225 (+1 ms)Running coprocessor post-close hooks at 1676037350228 (+3 ms)Closed at 1676037350228 2023-02-10 13:55:50,229 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,229 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=13a42a75712db718789d408f760bc81d, regionState=CLOSED 2023-02-10 13:55:50,230 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037350229"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037350229"}]},"ts":"1676037350229"} 2023-02-10 13:55:50,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=146, resume processing ppid=145 2023-02-10 13:55:50,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, ppid=145, state=SUCCESS; CloseRegionProcedure 13a42a75712db718789d408f760bc81d, server=jenkins-hbase4.apache.org,40651,1676037244010 in 190 msec 2023-02-10 13:55:50,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=144 2023-02-10 13:55:50,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=13a42a75712db718789d408f760bc81d, UNASSIGN in 194 msec 2023-02-10 13:55:50,241 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=144 splitting 14 storefiles, region=13a42a75712db718789d408f760bc81d, threads=14 2023-02-10 13:55:50,241 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,241 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,241 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,241 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,242 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,243 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,388 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,402 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,408 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,408 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,412 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,416 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,420 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,428 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,440 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,440 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,440 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,442 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,444 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,801 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52 for region: 13a42a75712db718789d408f760bc81d 2023-02-10 13:55:50,801 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=144 split storefiles for region 13a42a75712db718789d408f760bc81d Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-10 13:55:50,837 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:55:50,838 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-10 13:55:50,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1676037350839"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1676037350839"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1676037350839"}]},"ts":"1676037350839"} 2023-02-10 13:55:50,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037350839"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037350839"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037350839"}]},"ts":"1676037350839"} 2023-02-10 13:55:50,840 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037350839"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037350839"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037350839"}]},"ts":"1676037350839"} 2023-02-10 13:55:50,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, ASSIGN}, {pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, ASSIGN}] 2023-02-10 13:55:50,843 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, ASSIGN 2023-02-10 13:55:50,843 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:50,843 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, ASSIGN 2023-02-10 13:55:50,844 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,40651,1676037244010; forceNewPlan=false, retain=false 2023-02-10 13:55:50,993 INFO [jenkins-hbase4:38205] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-10 13:55:50,995 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=f3824dbfbd417a431cb43dca26b1a788, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:50,995 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037350995"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037350995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037350995"}]},"ts":"1676037350995"} 2023-02-10 13:55:50,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=147, state=RUNNABLE; OpenRegionProcedure f3824dbfbd417a431cb43dca26b1a788, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:50,998 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=c9538870cd651bd2f7ca1850fc413960, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:50,998 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037350998"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037350998"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037350998"}]},"ts":"1676037350998"} 2023-02-10 13:55:50,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=150, ppid=148, state=RUNNABLE; OpenRegionProcedure c9538870cd651bd2f7ca1850fc413960, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:51,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f3824dbfbd417a431cb43dca26b1a788, NAME => 'testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.', STARTKEY => '', ENDKEY => '41'} 2023-02-10 13:55:51,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:51,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:51,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,152 INFO [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,153 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1 2023-02-10 13:55:51,153 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1 2023-02-10 13:55:51,153 INFO [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3824dbfbd417a431cb43dca26b1a788 columnFamilyName cf1 2023-02-10 13:55:51,165 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52-bottom 2023-02-10 13:55:51,169 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188-bottom 2023-02-10 13:55:51,172 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699-bottom 2023-02-10 13:55:51,175 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73-bottom 2023-02-10 13:55:51,179 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b-bottom 2023-02-10 13:55:51,182 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded-bottom 2023-02-10 13:55:51,186 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119-bottom 2023-02-10 13:55:51,186 INFO [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(310): Store=f3824dbfbd417a431cb43dca26b1a788/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:51,186 INFO [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,187 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2 2023-02-10 13:55:51,187 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2 2023-02-10 13:55:51,187 INFO [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f3824dbfbd417a431cb43dca26b1a788 columnFamilyName cf2 2023-02-10 13:55:51,198 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f-bottom 2023-02-10 13:55:51,202 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211-bottom 2023-02-10 13:55:51,205 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb-bottom 2023-02-10 13:55:51,209 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16-bottom 2023-02-10 13:55:51,212 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0-bottom 2023-02-10 13:55:51,216 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a-bottom 2023-02-10 13:55:51,219 DEBUG [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023-bottom 2023-02-10 13:55:51,219 INFO [StoreOpener-f3824dbfbd417a431cb43dca26b1a788-1] regionserver.HStore(310): Store=f3824dbfbd417a431cb43dca26b1a788/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:51,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:51,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f3824dbfbd417a431cb43dca26b1a788; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58754333, jitterRate=-0.12449221312999725}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:51,223 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f3824dbfbd417a431cb43dca26b1a788: Running coprocessor pre-open hook at 1676037351151Writing region info on filesystem at 1676037351151Initializing all the Stores at 1676037351152 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037351152Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037351152Cleaning up temporary data from old regions at 1676037351221 (+69 ms)Cleaning up detritus from prior splits at 1676037351221Running coprocessor post-open hooks at 1676037351223 (+2 ms)Region opened successfully at 1676037351223 2023-02-10 13:55:51,224 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788., pid=149, masterSystemTime=1676037351148 2023-02-10 13:55:51,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-10 13:55:51,225 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:51,225 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:51,225 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,225 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,225 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): f3824dbfbd417a431cb43dca26b1a788/cf2 is initiating minor compaction (all files) 2023-02-10 13:55:51,225 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): f3824dbfbd417a431cb43dca26b1a788/cf1 is initiating minor compaction (all files) 2023-02-10 13:55:51,226 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of f3824dbfbd417a431cb43dca26b1a788/cf2 in testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,226 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of f3824dbfbd417a431cb43dca26b1a788/cf1 in testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,226 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/.tmp, totalSize=34.9 K 2023-02-10 13:55:51,226 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb-bottom] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/.tmp, totalSize=34.9 K 2023-02-10 13:55:51,226 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037349000 2023-02-10 13:55:51,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,227 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1676037349000 2023-02-10 13:55:51,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c9538870cd651bd2f7ca1850fc413960, NAME => 'testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.', STARTKEY => '41', ENDKEY => ''} 2023-02-10 13:55:51,227 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=f3824dbfbd417a431cb43dca26b1a788, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:51,227 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037349169 2023-02-10 13:55:51,227 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1676037349169 2023-02-10 13:55:51,227 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037351227"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037351227"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037351227"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037351227"}]},"ts":"1676037351227"} 2023-02-10 13:55:51,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-10 13:55:51,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-10 13:55:51,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,228 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037349349 2023-02-10 13:55:51,228 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1676037349349 2023-02-10 13:55:51,228 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037349518 2023-02-10 13:55:51,229 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1676037349518 2023-02-10 13:55:51,229 INFO [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,229 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037349694 2023-02-10 13:55:51,229 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1676037349694 2023-02-10 13:55:51,229 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037349863 2023-02-10 13:55:51,230 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1676037349863 2023-02-10 13:55:51,230 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037350029 2023-02-10 13:55:51,230 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=149, resume processing ppid=147 2023-02-10 13:55:51,230 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1 2023-02-10 13:55:51,230 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1 2023-02-10 13:55:51,230 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=147, state=SUCCESS; OpenRegionProcedure f3824dbfbd417a431cb43dca26b1a788, server=jenkins-hbase4.apache.org,40651,1676037244010 in 232 msec 2023-02-10 13:55:51,230 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1676037350029 2023-02-10 13:55:51,231 INFO [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9538870cd651bd2f7ca1850fc413960 columnFamilyName cf1 2023-02-10 13:55:51,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, ASSIGN in 388 msec 2023-02-10 13:55:51,241 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): f3824dbfbd417a431cb43dca26b1a788#cf1#compaction#142 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:51,243 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): f3824dbfbd417a431cb43dca26b1a788#cf2#compaction#143 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:51,243 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52-top 2023-02-10 13:55:51,246 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188-top 2023-02-10 13:55:51,262 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699-top 2023-02-10 13:55:51,264 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/.tmp/cf1/e5ecdf154ecb4f31b6500d65c4c0a120 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/e5ecdf154ecb4f31b6500d65c4c0a120 2023-02-10 13:55:51,264 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73-top 2023-02-10 13:55:51,267 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b-top 2023-02-10 13:55:51,274 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f3824dbfbd417a431cb43dca26b1a788/cf1 of f3824dbfbd417a431cb43dca26b1a788 into e5ecdf154ecb4f31b6500d65c4c0a120(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:51,274 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for f3824dbfbd417a431cb43dca26b1a788: 2023-02-10 13:55:51,274 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788., storeName=f3824dbfbd417a431cb43dca26b1a788/cf1, priority=9, startTime=1676037351224; duration=0sec 2023-02-10 13:55:51,274 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,281 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded-top 2023-02-10 13:55:51,283 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119-top 2023-02-10 13:55:51,283 INFO [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(310): Store=c9538870cd651bd2f7ca1850fc413960/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:51,284 INFO [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,284 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/.tmp/cf2/1ba5346e621b44b4b162c04687d85f8b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/1ba5346e621b44b4b162c04687d85f8b 2023-02-10 13:55:51,284 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2 2023-02-10 13:55:51,284 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2 2023-02-10 13:55:51,285 INFO [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9538870cd651bd2f7ca1850fc413960 columnFamilyName cf2 2023-02-10 13:55:51,289 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f3824dbfbd417a431cb43dca26b1a788/cf2 of f3824dbfbd417a431cb43dca26b1a788 into 1ba5346e621b44b4b162c04687d85f8b(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:51,289 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for f3824dbfbd417a431cb43dca26b1a788: 2023-02-10 13:55:51,289 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788., storeName=f3824dbfbd417a431cb43dca26b1a788/cf2, priority=9, startTime=1676037351225; duration=0sec 2023-02-10 13:55:51,289 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,296 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f-top 2023-02-10 13:55:51,298 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211-top 2023-02-10 13:55:51,300 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb-top 2023-02-10 13:55:51,303 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16-top 2023-02-10 13:55:51,306 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0-top 2023-02-10 13:55:51,309 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a-top 2023-02-10 13:55:51,311 DEBUG [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(539): loaded hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023-top 2023-02-10 13:55:51,312 INFO [StoreOpener-c9538870cd651bd2f7ca1850fc413960-1] regionserver.HStore(310): Store=c9538870cd651bd2f7ca1850fc413960/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-10 13:55:51,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-10 13:55:51,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c9538870cd651bd2f7ca1850fc413960; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73291061, jitterRate=0.0921219140291214}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-10 13:55:51,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c9538870cd651bd2f7ca1850fc413960: Running coprocessor pre-open hook at 1676037351228Writing region info on filesystem at 1676037351228Initializing all the Stores at 1676037351228Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037351228Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676037351228Cleaning up temporary data from old regions at 1676037351314 (+86 ms)Cleaning up detritus from prior splits at 1676037351314Running coprocessor post-open hooks at 1676037351316 (+2 ms)Region opened successfully at 1676037351316 2023-02-10 13:55:51,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960., pid=150, masterSystemTime=1676037351148 2023-02-10 13:55:51,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,317 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:51,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,317 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-10 13:55:51,318 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,318 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1912): c9538870cd651bd2f7ca1850fc413960/cf1 is initiating minor compaction (all files) 2023-02-10 13:55:51,318 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,318 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1912): c9538870cd651bd2f7ca1850fc413960/cf2 is initiating minor compaction (all files) 2023-02-10 13:55:51,318 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of c9538870cd651bd2f7ca1850fc413960/cf1 in testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,318 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2258): Starting compaction of c9538870cd651bd2f7ca1850fc413960/cf2 in testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,318 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b-top] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/.tmp, totalSize=34.9 K 2023-02-10 13:55:51,318 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb-top] into tmpdir=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/.tmp, totalSize=34.9 K 2023-02-10 13:55:51,318 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037349000 2023-02-10 13:55:51,319 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1676037349000 2023-02-10 13:55:51,319 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037349169 2023-02-10 13:55:51,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,319 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1676037349169 2023-02-10 13:55:51,319 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=c9538870cd651bd2f7ca1850fc413960, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:51,320 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037351319"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676037351319"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676037351319"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676037351319"}]},"ts":"1676037351319"} 2023-02-10 13:55:51,320 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037349349 2023-02-10 13:55:51,320 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1676037349349 2023-02-10 13:55:51,320 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037349518 2023-02-10 13:55:51,320 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1676037349518 2023-02-10 13:55:51,321 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1676037349694 2023-02-10 13:55:51,321 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1676037349694 2023-02-10 13:55:51,321 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting 04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1676037349863 2023-02-10 13:55:51,321 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1676037349863 2023-02-10 13:55:51,321 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] compactions.Compactor(207): Compacting eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037350029 2023-02-10 13:55:51,322 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] compactions.Compactor(207): Compacting 457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1676037350029 2023-02-10 13:55:51,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-02-10 13:55:51,323 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; OpenRegionProcedure c9538870cd651bd2f7ca1850fc413960, server=jenkins-hbase4.apache.org,40651,1676037244010 in 322 msec 2023-02-10 13:55:51,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=148, resume processing ppid=144 2023-02-10 13:55:51,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, ASSIGN in 481 msec 2023-02-10 13:55:51,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegion, parent=13a42a75712db718789d408f760bc81d, daughterA=f3824dbfbd417a431cb43dca26b1a788, daughterB=c9538870cd651bd2f7ca1850fc413960 in 1.2900 sec 2023-02-10 13:55:51,332 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] throttle.PressureAwareThroughputController(145): c9538870cd651bd2f7ca1850fc413960#cf2#compaction#145 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:51,332 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9538870cd651bd2f7ca1850fc413960#cf1#compaction#144 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-10 13:55:51,353 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/.tmp/cf1/fcb3c643b9894a92b92b31514bc81db6 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fcb3c643b9894a92b92b31514bc81db6 2023-02-10 13:55:51,355 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/.tmp/cf2/54e7105365174f09824470edf11b0ad5 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/54e7105365174f09824470edf11b0ad5 2023-02-10 13:55:51,357 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c9538870cd651bd2f7ca1850fc413960/cf1 of c9538870cd651bd2f7ca1850fc413960 into fcb3c643b9894a92b92b31514bc81db6(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:51,357 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for c9538870cd651bd2f7ca1850fc413960: 2023-02-10 13:55:51,357 INFO [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960., storeName=c9538870cd651bd2f7ca1850fc413960/cf1, priority=9, startTime=1676037351316; duration=0sec 2023-02-10 13:55:51,357 DEBUG [RS:2;jenkins-hbase4:40651-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,359 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c9538870cd651bd2f7ca1850fc413960/cf2 of c9538870cd651bd2f7ca1850fc413960 into 54e7105365174f09824470edf11b0ad5(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-10 13:55:51,359 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for c9538870cd651bd2f7ca1850fc413960: 2023-02-10 13:55:51,359 INFO [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960., storeName=c9538870cd651bd2f7ca1850fc413960/cf2, priority=9, startTime=1676037351317; duration=0sec 2023-02-10 13:55:51,359 DEBUG [RS:2;jenkins-hbase4:40651-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-10 13:55:51,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40651] ipc.CallRunner(144): callId: 1118 service: ClientService methodName: Scan size: 143 connection: 172.31.14.131:54866 deadline: 1676037411537, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d. is not online on jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:51,647 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:55:51,653 INFO [Listener at localhost/43477] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-10 13:55:51,654 DEBUG [Listener at localhost/43477] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-10 13:55:51,661 WARN [Listener at localhost/43477] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-10 13:55:51,662 INFO [Listener at localhost/43477] client.HBaseAdmin$15(890): Started disable of testSplitTableRegion 2023-02-10 13:55:51,662 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testSplitTableRegion 2023-02-10 13:55:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=151, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegion 2023-02-10 13:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-02-10 13:55:51,665 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037351665"}]},"ts":"1676037351665"} 2023-02-10 13:55:51,666 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLING in hbase:meta 2023-02-10 13:55:51,668 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegion to state=DISABLING 2023-02-10 13:55:51,669 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, UNASSIGN}, {pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, UNASSIGN}] 2023-02-10 13:55:51,670 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, UNASSIGN 2023-02-10 13:55:51,670 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=c9538870cd651bd2f7ca1850fc413960, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:51,670 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037351670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037351670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037351670"}]},"ts":"1676037351670"} 2023-02-10 13:55:51,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=154, ppid=153, state=RUNNABLE; CloseRegionProcedure c9538870cd651bd2f7ca1850fc413960, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:51,672 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, UNASSIGN 2023-02-10 13:55:51,673 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=f3824dbfbd417a431cb43dca26b1a788, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:51,673 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037351673"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676037351673"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676037351673"}]},"ts":"1676037351673"} 2023-02-10 13:55:51,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; CloseRegionProcedure f3824dbfbd417a431cb43dca26b1a788, server=jenkins-hbase4.apache.org,40651,1676037244010}] 2023-02-10 13:55:51,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-02-10 13:55:51,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f3824dbfbd417a431cb43dca26b1a788, disabling compactions & flushes 2023-02-10 13:55:51,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. after waiting 0 ms 2023-02-10 13:55:51,823 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,823 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b-bottom] to archive 2023-02-10 13:55:51,824 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:51,826 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,827 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,828 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,829 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,830 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,831 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,832 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,833 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211-bottom, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb-bottom] to archive 2023-02-10 13:55:51,833 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:51,834 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,835 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,836 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,837 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,838 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,839 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,840 DEBUG [StoreCloser-testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:55:51,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:51,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788. 2023-02-10 13:55:51,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f3824dbfbd417a431cb43dca26b1a788: Waiting for close lock at 1676037351823Running coprocessor pre-close hooks at 1676037351823Disabling compacts and flushes for region at 1676037351823Disabling writes for close at 1676037351823Writing region close event to WAL at 1676037351841 (+18 ms)Running coprocessor post-close hooks at 1676037351843 (+2 ms)Closed at 1676037351844 (+1 ms) 2023-02-10 13:55:51,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c9538870cd651bd2f7ca1850fc413960, disabling compactions & flushes 2023-02-10 13:55:51,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. after waiting 0 ms 2023-02-10 13:55:51,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,845 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=f3824dbfbd417a431cb43dca26b1a788, regionState=CLOSED 2023-02-10 13:55:51,845 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037351845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037351845"}]},"ts":"1676037351845"} 2023-02-10 13:55:51,846 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b-top] to archive 2023-02-10 13:55:51,847 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:51,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-02-10 13:55:51,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; CloseRegionProcedure f3824dbfbd417a431cb43dca26b1a788, server=jenkins-hbase4.apache.org,40651,1676037244010 in 173 msec 2023-02-10 13:55:51,849 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/7697412c107c46349bfb4ee5f8b98188.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=f3824dbfbd417a431cb43dca26b1a788, UNASSIGN in 180 msec 2023-02-10 13:55:51,850 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/d229eaf9fdba499cb622e3f24c864d73.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,851 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/feb699dbd8d143d98ffeaa8039c36119.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,852 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/ae3d6bedbeea4f578c5493040838c699.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,853 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fb218491ada9461e82e255c50641dded.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,854 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/04ed82e45a3d4d208585210b600f4c52.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,855 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/eda996ff76dd4781a7df432fba7c3c0b.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,857 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211-top, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d->hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb-top] to archive 2023-02-10 13:55:51,857 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-10 13:55:51,859 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/90a65bbfb21e4c4183fe699570c6ed16.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,860 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/db86f397877940529c57ab9879b3020a.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,861 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/ef7e77931b35455e8656fa73e940c023.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,862 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/06a28eca1f2e43db8015268fd700ec1f.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,863 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/d837d995b5b242c98d15ebd66d23e1f0.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,864 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/2d3d5e3d01d748f6925875a267062211.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,865 DEBUG [StoreCloser-testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb.13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-10 13:55:51,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:51,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960. 2023-02-10 13:55:51,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c9538870cd651bd2f7ca1850fc413960: Waiting for close lock at 1676037351845Running coprocessor pre-close hooks at 1676037351845Disabling compacts and flushes for region at 1676037351845Disabling writes for close at 1676037351845Writing region close event to WAL at 1676037351866 (+21 ms)Running coprocessor post-close hooks at 1676037351869 (+3 ms)Closed at 1676037351869 2023-02-10 13:55:51,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,871 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=c9538870cd651bd2f7ca1850fc413960, regionState=CLOSED 2023-02-10 13:55:51,871 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1676037351871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676037351871"}]},"ts":"1676037351871"} 2023-02-10 13:55:51,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=153 2023-02-10 13:55:51,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=153, state=SUCCESS; CloseRegionProcedure c9538870cd651bd2f7ca1850fc413960, server=jenkins-hbase4.apache.org,40651,1676037244010 in 201 msec 2023-02-10 13:55:51,875 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=151 2023-02-10 13:55:51,875 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=c9538870cd651bd2f7ca1850fc413960, UNASSIGN in 204 msec 2023-02-10 13:55:51,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676037351875"}]},"ts":"1676037351875"} 2023-02-10 13:55:51,876 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLED in hbase:meta 2023-02-10 13:55:51,878 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegion to state=DISABLED 2023-02-10 13:55:51,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, state=SUCCESS; DisableTableProcedure table=testSplitTableRegion in 216 msec 2023-02-10 13:55:51,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-02-10 13:55:51,967 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegion, procId: 151 completed 2023-02-10 13:55:51,967 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testSplitTableRegion 2023-02-10 13:55:51,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] procedure2.ProcedureExecutor(1029): Stored pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegion 2023-02-10 13:55:51,970 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-10 13:55:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-02-10 13:55:51,970 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=156, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-10 13:55:51,974 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,974 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,974 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,976 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/recovered.edits] 2023-02-10 13:55:51,976 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/recovered.edits] 2023-02-10 13:55:51,976 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2, FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/recovered.edits] 2023-02-10 13:55:51,979 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fcb3c643b9894a92b92b31514bc81db6 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf1/fcb3c643b9894a92b92b31514bc81db6 2023-02-10 13:55:51,979 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/e5ecdf154ecb4f31b6500d65c4c0a120 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf1/e5ecdf154ecb4f31b6500d65c4c0a120 2023-02-10 13:55:51,980 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/04ed82e45a3d4d208585210b600f4c52 2023-02-10 13:55:51,981 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/7697412c107c46349bfb4ee5f8b98188 2023-02-10 13:55:51,981 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/54e7105365174f09824470edf11b0ad5 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/cf2/54e7105365174f09824470edf11b0ad5 2023-02-10 13:55:51,982 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/1ba5346e621b44b4b162c04687d85f8b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/cf2/1ba5346e621b44b4b162c04687d85f8b 2023-02-10 13:55:51,982 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/ae3d6bedbeea4f578c5493040838c699 2023-02-10 13:55:51,984 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/d229eaf9fdba499cb622e3f24c864d73 2023-02-10 13:55:51,984 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960/recovered.edits/90.seqid 2023-02-10 13:55:51,984 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/recovered.edits/90.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788/recovered.edits/90.seqid 2023-02-10 13:55:51,985 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/c9538870cd651bd2f7ca1850fc413960 2023-02-10 13:55:51,985 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/f3824dbfbd417a431cb43dca26b1a788 2023-02-10 13:55:51,985 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/eda996ff76dd4781a7df432fba7c3c0b 2023-02-10 13:55:51,986 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/fb218491ada9461e82e255c50641dded 2023-02-10 13:55:51,987 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf1/feb699dbd8d143d98ffeaa8039c36119 2023-02-10 13:55:51,990 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/06a28eca1f2e43db8015268fd700ec1f 2023-02-10 13:55:51,991 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/2d3d5e3d01d748f6925875a267062211 2023-02-10 13:55:51,991 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/457eddeec19f4d4c9e1d9e8e4de7dbeb 2023-02-10 13:55:51,992 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/90a65bbfb21e4c4183fe699570c6ed16 2023-02-10 13:55:51,993 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/d837d995b5b242c98d15ebd66d23e1f0 2023-02-10 13:55:51,993 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/db86f397877940529c57ab9879b3020a 2023-02-10 13:55:51,994 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023 to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/cf2/ef7e77931b35455e8656fa73e940c023 2023-02-10 13:55:51,996 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/recovered.edits/85.seqid to hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/archive/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d/recovered.edits/85.seqid 2023-02-10 13:55:51,996 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/.tmp/data/default/testSplitTableRegion/13a42a75712db718789d408f760bc81d 2023-02-10 13:55:51,996 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-02-10 13:55:51,998 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=156, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-10 13:55:51,999 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegion from hbase:meta 2023-02-10 13:55:52,000 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegion' descriptor. 2023-02-10 13:55:52,001 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=156, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-10 13:55:52,001 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegion' from region states. 2023-02-10 13:55:52,001 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037352001"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:52,001 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037352001"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:52,001 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676037352001"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:52,003 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-10 13:55:52,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 13a42a75712db718789d408f760bc81d, NAME => 'testSplitTableRegion,,1676037348497.13a42a75712db718789d408f760bc81d.', STARTKEY => '', ENDKEY => ''}, {ENCODED => f3824dbfbd417a431cb43dca26b1a788, NAME => 'testSplitTableRegion,,1676037350035.f3824dbfbd417a431cb43dca26b1a788.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => c9538870cd651bd2f7ca1850fc413960, NAME => 'testSplitTableRegion,41,1676037350035.c9538870cd651bd2f7ca1850fc413960.', STARTKEY => '41', ENDKEY => ''}] 2023-02-10 13:55:52,003 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegion' as deleted. 2023-02-10 13:55:52,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676037352003"}]},"ts":"9223372036854775807"} 2023-02-10 13:55:52,004 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegion state from META 2023-02-10 13:55:52,006 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=156, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-10 13:55:52,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=156, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegion in 39 msec 2023-02-10 13:55:52,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38205] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-02-10 13:55:52,071 INFO [Listener at localhost/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegion, procId: 156 completed 2023-02-10 13:55:52,085 INFO [Listener at localhost/43477] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=374 (was 373) - Thread LEAK? -, OpenFileDescriptor=834 (was 834), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=72 (was 70) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3126 (was 3137) 2023-02-10 13:55:52,085 INFO [Listener at localhost/43477] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-10 13:55:52,085 INFO [Listener at localhost/43477] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-10 13:55:52,085 DEBUG [Listener at localhost/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ff4e667 to 127.0.0.1:63999 2023-02-10 13:55:52,085 DEBUG [Listener at localhost/43477] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,086 DEBUG [Listener at localhost/43477] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-10 13:55:52,086 DEBUG [Listener at localhost/43477] util.JVMClusterUtil(257): Found active master hash=465490524, stopped=false 2023-02-10 13:55:52,086 INFO [Listener at localhost/43477] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:55:52,088 INFO [Listener at localhost/43477] procedure2.ProcedureExecutor(629): Stopping 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70bdf07e to 127.0.0.1:63999 2023-02-10 13:55:52,088 DEBUG [Listener at localhost/43477] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:55:52,089 INFO [Listener at localhost/43477] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39591,1676037243910' ***** 2023-02-10 13:55:52,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:55:52,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:55:52,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-10 13:55:52,089 INFO [Listener at localhost/43477] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-10 13:55:52,089 INFO [Listener at localhost/43477] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46307,1676037243975' ***** 2023-02-10 13:55:52,089 INFO [Listener at localhost/43477] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-10 13:55:52,089 INFO [Listener at localhost/43477] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40651,1676037244010' ***** 2023-02-10 13:55:52,089 INFO [Listener at localhost/43477] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-10 13:55:52,089 INFO [RS:2;jenkins-hbase4:40651] regionserver.HeapMemoryManager(220): Stopping 2023-02-10 13:55:52,089 INFO [RS:0;jenkins-hbase4:39591] regionserver.HeapMemoryManager(220): Stopping 2023-02-10 13:55:52,089 INFO [RS:1;jenkins-hbase4:46307] regionserver.HeapMemoryManager(220): Stopping 2023-02-10 13:55:52,090 INFO [RS:2;jenkins-hbase4:40651] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-10 13:55:52,090 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-10 13:55:52,090 INFO [RS:2;jenkins-hbase4:40651] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-10 13:55:52,090 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-10 13:55:52,090 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:52,090 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-10 13:55:52,090 DEBUG [RS:2;jenkins-hbase4:40651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fd7b187 to 127.0.0.1:63999 2023-02-10 13:55:52,090 DEBUG [RS:2;jenkins-hbase4:40651] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,090 INFO [RS:0;jenkins-hbase4:39591] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-10 13:55:52,090 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40651,1676037244010; all regions closed. 2023-02-10 13:55:52,090 INFO [RS:0;jenkins-hbase4:39591] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-10 13:55:52,090 INFO [RS:1;jenkins-hbase4:46307] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-10 13:55:52,090 INFO [RS:1;jenkins-hbase4:46307] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-10 13:55:52,090 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(3304): Received CLOSE for 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:55:52,090 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:52,090 DEBUG [RS:1;jenkins-hbase4:46307] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fc258b9 to 127.0.0.1:63999 2023-02-10 13:55:52,090 DEBUG [RS:1;jenkins-hbase4:46307] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,091 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46307,1676037243975; all regions closed. 2023-02-10 13:55:52,091 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:52,091 DEBUG [RS:0;jenkins-hbase4:39591] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19aceb18 to 127.0.0.1:63999 2023-02-10 13:55:52,091 DEBUG [RS:0;jenkins-hbase4:39591] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 68b3720325c3cd17f2ad6817c681aaf1, disabling compactions & flushes 2023-02-10 13:55:52,091 INFO [RS:0;jenkins-hbase4:39591] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-10 13:55:52,091 INFO [RS:0;jenkins-hbase4:39591] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-10 13:55:52,091 INFO [RS:0;jenkins-hbase4:39591] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-10 13:55:52,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:55:52,091 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-10 13:55:52,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:55:52,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. after waiting 0 ms 2023-02-10 13:55:52,091 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:55:52,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 68b3720325c3cd17f2ad6817c681aaf1 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-10 13:55:52,091 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-10 13:55:52,091 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-10 13:55:52,091 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 68b3720325c3cd17f2ad6817c681aaf1=hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1.} 2023-02-10 13:55:52,091 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-10 13:55:52,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-10 13:55:52,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-10 13:55:52,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-10 13:55:52,092 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=79.40 KB heapSize=125.77 KB 2023-02-10 13:55:52,092 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1505): Waiting on 1588230740, 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:55:52,107 DEBUG [RS:2;jenkins-hbase4:40651] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs 2023-02-10 13:55:52,107 INFO [RS:2;jenkins-hbase4:40651] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40651%2C1676037244010:(num 1676037245607) 2023-02-10 13:55:52,107 DEBUG [RS:2;jenkins-hbase4:40651] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,108 INFO [RS:2;jenkins-hbase4:40651] regionserver.LeaseManager(133): Closed leases 2023-02-10 13:55:52,113 INFO [RS:2;jenkins-hbase4:40651] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-10 13:55:52,113 INFO [RS:2;jenkins-hbase4:40651] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-10 13:55:52,113 DEBUG [RS:1;jenkins-hbase4:46307] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs 2023-02-10 13:55:52,113 INFO [RS:2;jenkins-hbase4:40651] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-10 13:55:52,113 INFO [RS:2;jenkins-hbase4:40651] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-10 13:55:52,113 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-10 13:55:52,113 INFO [RS:1;jenkins-hbase4:46307] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46307%2C1676037243975:(num 1676037245607) 2023-02-10 13:55:52,114 DEBUG [RS:1;jenkins-hbase4:46307] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,114 INFO [RS:1;jenkins-hbase4:46307] regionserver.LeaseManager(133): Closed leases 2023-02-10 13:55:52,114 INFO [RS:2;jenkins-hbase4:40651] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40651 2023-02-10 13:55:52,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/.tmp/info/58f2131f26964b31a0f477fa48b0732e 2023-02-10 13:55:52,269 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-10 13:55:52,269 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-10 13:55:52,269 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-10 13:55:52,270 INFO [RS:1;jenkins-hbase4:46307] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-10 13:55:52,271 INFO [RS:1;jenkins-hbase4:46307] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-10 13:55:52,271 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-10 13:55:52,271 INFO [RS:1;jenkins-hbase4:46307] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-10 13:55:52,271 INFO [RS:1;jenkins-hbase4:46307] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-10 13:55:52,272 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:52,272 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:55:52,272 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:52,272 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:55:52,273 INFO [RS:1;jenkins-hbase4:46307] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46307 2023-02-10 13:55:52,273 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3a7185b4 rejected from java.util.concurrent.ThreadPoolExecutor@3e4e1622[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,274 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40651,1676037244010] 2023-02-10 13:55:52,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40651,1676037244010 2023-02-10 13:55:52,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:55:52,274 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40651,1676037244010; numProcessing=1 2023-02-10 13:55:52,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:55:52,274 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c972159 rejected from java.util.concurrent.ThreadPoolExecutor@3e4e1622[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:52,274 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:52,274 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4cca721 rejected from java.util.concurrent.ThreadPoolExecutor@3e4e1622[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,276 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46307,1676037243975 2023-02-10 13:55:52,276 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:55:52,276 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e57e510 rejected from java.util.concurrent.ThreadPoolExecutor@a686002[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,276 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.29 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/.tmp/info/73baca6dde4f4fd2a8d0b4279af07ff2 2023-02-10 13:55:52,277 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40651,1676037244010 already deleted, retry=false 2023-02-10 13:55:52,277 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40651,1676037244010 expired; onlineServers=2 2023-02-10 13:55:52,279 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46307,1676037243975] 2023-02-10 13:55:52,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/.tmp/info/58f2131f26964b31a0f477fa48b0732e as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/info/58f2131f26964b31a0f477fa48b0732e 2023-02-10 13:55:52,279 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46307,1676037243975; numProcessing=2 2023-02-10 13:55:52,282 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46307,1676037243975 already deleted, retry=false 2023-02-10 13:55:52,282 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46307,1676037243975 expired; onlineServers=1 2023-02-10 13:55:52,282 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 73baca6dde4f4fd2a8d0b4279af07ff2 2023-02-10 13:55:52,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/info/58f2131f26964b31a0f477fa48b0732e, entries=2, sequenceid=6, filesize=4.8 K 2023-02-10 13:55:52,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 68b3720325c3cd17f2ad6817c681aaf1 in 197ms, sequenceid=6, compaction requested=false 2023-02-10 13:55:52,292 DEBUG [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1505): Waiting on 1588230740, 68b3720325c3cd17f2ad6817c681aaf1 2023-02-10 13:55:52,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/namespace/68b3720325c3cd17f2ad6817c681aaf1/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-10 13:55:52,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:52,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:55:52,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 68b3720325c3cd17f2ad6817c681aaf1: Waiting for close lock at 1676037352091Running coprocessor pre-close hooks at 1676037352091Disabling compacts and flushes for region at 1676037352091Disabling writes for close at 1676037352091Obtaining lock to block concurrent updates at 1676037352091Preparing flush snapshotting stores in 68b3720325c3cd17f2ad6817c681aaf1 at 1676037352091Finished memstore snapshotting hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676037352092 (+1 ms)Flushing stores of hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. at 1676037352093 (+1 ms)Flushing 68b3720325c3cd17f2ad6817c681aaf1/info: creating writer at 1676037352093Flushing 68b3720325c3cd17f2ad6817c681aaf1/info: appending metadata at 1676037352101 (+8 ms)Flushing 68b3720325c3cd17f2ad6817c681aaf1/info: closing flushed file at 1676037352101Flushing 68b3720325c3cd17f2ad6817c681aaf1/info: reopening flushed file at 1676037352282 (+181 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 68b3720325c3cd17f2ad6817c681aaf1 in 197ms, sequenceid=6, compaction requested=false at 1676037352288 (+6 ms)Writing region close event to WAL at 1676037352289 (+1 ms)Running coprocessor post-close hooks at 1676037352294 (+5 ms)Closed at 1676037352295 (+1 ms) 2023-02-10 13:55:52,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676037246074.68b3720325c3cd17f2ad6817c681aaf1. 2023-02-10 13:55:52,296 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.58 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/.tmp/rep_barrier/27bf36ea1d06427f8ee4ba9dc0c9de64 2023-02-10 13:55:52,299 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 27bf36ea1d06427f8ee4ba9dc0c9de64 2023-02-10 13:55:52,312 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=5.53 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/.tmp/table/abe69c4df6784cbb9b05f08e7e70628b 2023-02-10 13:55:52,315 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for abe69c4df6784cbb9b05f08e7e70628b 2023-02-10 13:55:52,315 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/.tmp/info/73baca6dde4f4fd2a8d0b4279af07ff2 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/info/73baca6dde4f4fd2a8d0b4279af07ff2 2023-02-10 13:55:52,318 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 73baca6dde4f4fd2a8d0b4279af07ff2 2023-02-10 13:55:52,318 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/info/73baca6dde4f4fd2a8d0b4279af07ff2, entries=59, sequenceid=210, filesize=11.1 K 2023-02-10 13:55:52,319 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/.tmp/rep_barrier/27bf36ea1d06427f8ee4ba9dc0c9de64 as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/rep_barrier/27bf36ea1d06427f8ee4ba9dc0c9de64 2023-02-10 13:55:52,322 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 27bf36ea1d06427f8ee4ba9dc0c9de64 2023-02-10 13:55:52,322 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/rep_barrier/27bf36ea1d06427f8ee4ba9dc0c9de64, entries=24, sequenceid=210, filesize=7.5 K 2023-02-10 13:55:52,322 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/.tmp/table/abe69c4df6784cbb9b05f08e7e70628b as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/table/abe69c4df6784cbb9b05f08e7e70628b 2023-02-10 13:55:52,325 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for abe69c4df6784cbb9b05f08e7e70628b 2023-02-10 13:55:52,325 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/table/abe69c4df6784cbb9b05f08e7e70628b, entries=36, sequenceid=210, filesize=8.0 K 2023-02-10 13:55:52,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~79.40 KB/81308, heapSize ~125.73 KB/128744, currentSize=0 B/0 for 1588230740 in 234ms, sequenceid=210, compaction requested=false 2023-02-10 13:55:52,330 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/data/hbase/meta/1588230740/recovered.edits/213.seqid, newMaxSeqId=213, maxSeqId=1 2023-02-10 13:55:52,330 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:52,330 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-10 13:55:52,331 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-10 13:55:52,331 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676037352091Running coprocessor pre-close hooks at 1676037352091Disabling compacts and flushes for region at 1676037352091Disabling writes for close at 1676037352092 (+1 ms)Obtaining lock to block concurrent updates at 1676037352092Preparing flush snapshotting stores in 1588230740 at 1676037352092Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=81308, getHeapSize=128744, getOffHeapSize=0, getCellsCount=617 at 1676037352092Flushing stores of hbase:meta,,1.1588230740 at 1676037352093 (+1 ms)Flushing 1588230740/info: creating writer at 1676037352093Flushing 1588230740/info: appending metadata at 1676037352103 (+10 ms)Flushing 1588230740/info: closing flushed file at 1676037352103Flushing 1588230740/rep_barrier: creating writer at 1676037352283 (+180 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676037352285 (+2 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676037352285Flushing 1588230740/table: creating writer at 1676037352299 (+14 ms)Flushing 1588230740/table: appending metadata at 1676037352301 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676037352301Flushing 1588230740/info: reopening flushed file at 1676037352316 (+15 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676037352319 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676037352323 (+4 ms)Finished flush of dataSize ~79.40 KB/81308, heapSize ~125.73 KB/128744, currentSize=0 B/0 for 1588230740 in 234ms, sequenceid=210, compaction requested=false at 1676037352326 (+3 ms)Writing region close event to WAL at 1676037352328 (+2 ms)Running coprocessor post-close hooks at 1676037352330 (+2 ms)Closed at 1676037352331 (+1 ms) 2023-02-10 13:55:52,331 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-10 13:55:52,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,376 INFO [RS:2;jenkins-hbase4:40651] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40651,1676037244010; zookeeper connection closed. 2023-02-10 13:55:52,376 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5e3b071c rejected from java.util.concurrent.ThreadPoolExecutor@3e4e1622[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,376 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:40651-0x1003e82a6100003, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,376 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1fc88953 rejected from java.util.concurrent.ThreadPoolExecutor@3e4e1622[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,376 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@d9ca4c9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@d9ca4c9 2023-02-10 13:55:52,448 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-10 13:55:52,448 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-10 13:55:52,476 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,476 INFO [RS:1;jenkins-hbase4:46307] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46307,1676037243975; zookeeper connection closed. 2023-02-10 13:55:52,476 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4fa1b62f rejected from java.util.concurrent.ThreadPoolExecutor@a686002[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,477 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3b97165c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3b97165c 2023-02-10 13:55:52,477 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:46307-0x1003e82a6100002, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,477 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@374ea211 rejected from java.util.concurrent.ThreadPoolExecutor@a686002[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,493 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39591,1676037243910; all regions closed. 2023-02-10 13:55:52,498 DEBUG [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs 2023-02-10 13:55:52,498 INFO [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39591%2C1676037243910.meta:.meta(num 1676037245811) 2023-02-10 13:55:52,501 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/WALs/jenkins-hbase4.apache.org,39591,1676037243910/jenkins-hbase4.apache.org%2C39591%2C1676037243910.1676037245607 not finished, retry = 0 2023-02-10 13:55:52,582 ERROR [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1527945586-172.31.14.131-1676037238735 (Datanode Uuid 3707987e-c8ef-479c-bbb7-c70ee662bee7) service to localhost/127.0.0.1:36875 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-10 13:55:52,603 DEBUG [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/oldWALs 2023-02-10 13:55:52,603 INFO [RS:0;jenkins-hbase4:39591] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39591%2C1676037243910:(num 1676037245607) 2023-02-10 13:55:52,603 DEBUG [RS:0;jenkins-hbase4:39591] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,603 INFO [RS:0;jenkins-hbase4:39591] regionserver.LeaseManager(133): Closed leases 2023-02-10 13:55:52,604 INFO [RS:0;jenkins-hbase4:39591] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-10 13:55:52,604 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-10 13:55:52,605 INFO [RS:0;jenkins-hbase4:39591] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39591 2023-02-10 13:55:52,606 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39591,1676037243910 2023-02-10 13:55:52,606 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-10 13:55:52,606 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b8f0c0b rejected from java.util.concurrent.ThreadPoolExecutor@2fb09da2[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,607 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39591,1676037243910] 2023-02-10 13:55:52,607 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39591,1676037243910; numProcessing=3 2023-02-10 13:55:52,609 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39591,1676037243910 already deleted, retry=false 2023-02-10 13:55:52,609 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39591,1676037243910 expired; onlineServers=0 2023-02-10 13:55:52,609 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38205,1676037242772' ***** 2023-02-10 13:55:52,609 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-10 13:55:52,610 DEBUG [M:0;jenkins-hbase4:38205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a83d867, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-10 13:55:52,610 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38205,1676037242772 2023-02-10 13:55:52,611 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38205,1676037242772; all regions closed. 2023-02-10 13:55:52,611 DEBUG [M:0;jenkins-hbase4:38205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-10 13:55:52,611 DEBUG [M:0;jenkins-hbase4:38205] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-10 13:55:52,611 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-10 13:55:52,611 DEBUG [M:0;jenkins-hbase4:38205] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-10 13:55:52,611 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676037245183] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676037245183,5,FailOnTimeoutGroup] 2023-02-10 13:55:52,611 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676037245180] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676037245180,5,FailOnTimeoutGroup] 2023-02-10 13:55:52,611 INFO [M:0;jenkins-hbase4:38205] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-10 13:55:52,611 INFO [M:0;jenkins-hbase4:38205] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-10 13:55:52,611 INFO [M:0;jenkins-hbase4:38205] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-10 13:55:52,611 DEBUG [M:0;jenkins-hbase4:38205] master.HMaster(1502): Stopping service threads 2023-02-10 13:55:52,611 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-10 13:55:52,611 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-10 13:55:52,611 INFO [M:0;jenkins-hbase4:38205] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-10 13:55:52,612 ERROR [M:0;jenkins-hbase4:38205] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] Thread[HFileArchiver-14,5,PEWorkerGroup] Thread[HFileArchiver-15,5,PEWorkerGroup] Thread[HFileArchiver-16,5,PEWorkerGroup] 2023-02-10 13:55:52,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-10 13:55:52,612 INFO [M:0;jenkins-hbase4:38205] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-10 13:55:52,612 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-10 13:55:52,612 DEBUG [M:0;jenkins-hbase4:38205] zookeeper.ZKUtil(398): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-10 13:55:52,612 WARN [M:0;jenkins-hbase4:38205] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-10 13:55:52,613 INFO [M:0;jenkins-hbase4:38205] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-10 13:55:52,613 INFO [M:0;jenkins-hbase4:38205] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-10 13:55:52,613 DEBUG [M:0;jenkins-hbase4:38205] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-10 13:55:52,613 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:55:52,613 DEBUG [M:0;jenkins-hbase4:38205] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:55:52,613 DEBUG [M:0;jenkins-hbase4:38205] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-10 13:55:52,613 DEBUG [M:0;jenkins-hbase4:38205] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:55:52,613 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=608.14 KB heapSize=719.05 KB 2023-02-10 13:55:52,623 INFO [M:0;jenkins-hbase4:38205] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=608.14 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d697f3d011354f0c9a0cdc99c76293cf 2023-02-10 13:55:52,625 INFO [M:0;jenkins-hbase4:38205] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d697f3d011354f0c9a0cdc99c76293cf 2023-02-10 13:55:52,626 DEBUG [M:0;jenkins-hbase4:38205] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d697f3d011354f0c9a0cdc99c76293cf as hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d697f3d011354f0c9a0cdc99c76293cf 2023-02-10 13:55:52,628 INFO [M:0;jenkins-hbase4:38205] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d697f3d011354f0c9a0cdc99c76293cf 2023-02-10 13:55:52,629 INFO [M:0;jenkins-hbase4:38205] regionserver.HStore(1080): Added hdfs://localhost:36875/user/jenkins/test-data/84873e5c-41ef-4e09-ac11-ce10e793e963/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d697f3d011354f0c9a0cdc99c76293cf, entries=156, sequenceid=1282, filesize=34.9 K 2023-02-10 13:55:52,629 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegion(2947): Finished flush of dataSize ~608.14 KB/622734, heapSize ~719.04 KB/736296, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 16ms, sequenceid=1282, compaction requested=false 2023-02-10 13:55:52,630 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-10 13:55:52,630 DEBUG [M:0;jenkins-hbase4:38205] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676037352613Disabling compacts and flushes for region at 1676037352613Disabling writes for close at 1676037352613Obtaining lock to block concurrent updates at 1676037352613Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676037352613Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=622734, getHeapSize=736296, getOffHeapSize=0, getCellsCount=1499 at 1676037352613Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676037352614 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676037352614Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676037352617 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676037352617Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676037352626 (+9 ms)Finished flush of dataSize ~608.14 KB/622734, heapSize ~719.04 KB/736296, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 16ms, sequenceid=1282, compaction requested=false at 1676037352629 (+3 ms)Writing region close event to WAL at 1676037352630 (+1 ms)Closed at 1676037352630 2023-02-10 13:55:52,633 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-10 13:55:52,633 INFO [M:0;jenkins-hbase4:38205] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-10 13:55:52,634 INFO [M:0;jenkins-hbase4:38205] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38205 2023-02-10 13:55:52,635 DEBUG [M:0;jenkins-hbase4:38205] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38205,1676037242772 already deleted, retry=false 2023-02-10 13:55:52,708 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,708 INFO [RS:0;jenkins-hbase4:39591] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39591,1676037243910; zookeeper connection closed. 2023-02-10 13:55:52,708 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@82e83d2 rejected from java.util.concurrent.ThreadPoolExecutor@2fb09da2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,708 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39591-0x1003e82a6100001, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,708 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6230b171] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6230b171 2023-02-10 13:55:52,708 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7064f2c5 rejected from java.util.concurrent.ThreadPoolExecutor@2fb09da2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,708 INFO [Listener at localhost/43477] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-10 13:55:52,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,808 INFO [M:0;jenkins-hbase4:38205] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38205,1676037242772; zookeeper connection closed. 2023-02-10 13:55:52,808 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@101170af rejected from java.util.concurrent.ThreadPoolExecutor@32fcc324[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 753] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,808 DEBUG [Listener at localhost/43477-EventThread] zookeeper.ZKWatcher(600): master:38205-0x1003e82a6100000, quorum=127.0.0.1:63999, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-10 13:55:52,808 ERROR [Listener at localhost/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@66c626e9 rejected from java.util.concurrent.ThreadPoolExecutor@32fcc324[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 753] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-10 13:55:52,810 WARN [Listener at localhost/43477] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-10 13:55:52,814 INFO [Listener at localhost/43477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-10 13:55:52,920 WARN [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-10 13:55:52,920 WARN [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1527945586-172.31.14.131-1676037238735 (Datanode Uuid 7cac6cd9-03f4-49d1-b43b-1428b7e6fbc3) service to localhost/127.0.0.1:36875 2023-02-10 13:55:52,922 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data5/current/BP-1527945586-172.31.14.131-1676037238735] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-10 13:55:52,923 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data6/current/BP-1527945586-172.31.14.131-1676037238735] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-10 13:55:52,924 WARN [Listener at localhost/43477] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-10 13:55:52,927 INFO [Listener at localhost/43477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-10 13:55:53,032 WARN [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1527945586-172.31.14.131-1676037238735 (Datanode Uuid 3707987e-c8ef-479c-bbb7-c70ee662bee7) service to localhost/127.0.0.1:36875 2023-02-10 13:55:53,035 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data3/current/BP-1527945586-172.31.14.131-1676037238735] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-10 13:55:53,035 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data4/current/BP-1527945586-172.31.14.131-1676037238735] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-10 13:55:53,037 WARN [Listener at localhost/43477] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-10 13:55:53,039 INFO [Listener at localhost/43477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-10 13:55:53,139 WARN [BP-1527945586-172.31.14.131-1676037238735 heartbeating to localhost/127.0.0.1:36875] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1527945586-172.31.14.131-1676037238735 (Datanode Uuid 10b8853b-22ce-4aa3-86bd-2e3e9072bae3) service to localhost/127.0.0.1:36875 2023-02-10 13:55:53,140 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data1/current/BP-1527945586-172.31.14.131-1676037238735] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-10 13:55:53,141 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b6ec316d-989e-11b4-6063-bde10303ca9c/cluster_baf91bb8-37ba-d912-5e3f-072fd4188ca8/dfs/data/data2/current/BP-1527945586-172.31.14.131-1676037238735] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-10 13:55:53,162 INFO [Listener at localhost/43477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-10 13:55:53,324 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-10 13:55:53,357 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-10 13:55:53,358 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-10 13:55:53,361 INFO [Listener at localhost/43477] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-10 13:55:53,601 INFO [Listener at localhost/43477] hbase.HBaseTestingUtility(1293): Minicluster is down